CatalogLoader 2014 Belarus Minsk DNUG
-
Upload
nikolai-kekish -
Category
Technology
-
view
110 -
download
6
Transcript of CatalogLoader 2014 Belarus Minsk DNUG
![Page 1: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/1.jpg)
Николай Кекиш Минск - 2014
Как сделать 1000 парсеров и остаться довольным?
CatalogLoader.com
![Page 2: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/2.jpg)
Слайд 2 из 22
О компании
- 6 продуктов- 30 CMS- 500 проектов- 2500 клиентов
CatalogLoader.com
![Page 3: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/3.jpg)
Слайд 3 из 22
Парсер извлекает
- категории \ продукты- параметры \ комбинации - картинки
![Page 4: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/4.jpg)
Слайд 4 из 22
История
- просто код
- проект- скрипты- версии
![Page 5: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/5.jpg)
Слайд 5 из 22
HtmlAgilityPackvar doc = new HtmlDocument();
doc.Load("file.htm");
var attributes = new List<HtmlAttribute>();
foreach(HtmlNode link in doc.DocumentElement.SelectNodes("//a[@href"])
{
var att = link["href"];
attributes.Add(att);
}
![Page 6: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/6.jpg)
Слайд 6 из 22
Проект (настройки) <GrabberSettings>
<CategoryLevel1 />
<CategoryLevel2 />
<CategoryLevel3 />
<ShopUrl>http://artans.ru</ShopUrl>
<TypeOfHrefForProducts />
<Charset />
<PaginationTemplate>index.php?page={0}</PaginationTemplate>
<ProductsOnPage>1</ProductsOnPage>
<MaxProductsInCategory>0</MaxProductsInCategory>
<PaginationStartIndex>1</PaginationStartIndex>
![Page 7: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/7.jpg)
Слайд 7 из 22
Проект (каталог)#Men[path]http://www.thehut.com/catalogue/men.list@@1000
##New In[path]http://www.thehut.com/catalogue/men/new-in.list@@1011
###Hoody[path]http://www.thehut.com/catalogue/men/new-in.list?searchFilters=SG9vZHkjQ2xvdGhpbmd0eXBlX2NvbnRlbnQ6SG9vZHkhUmZXN0cmljdGVkOmZhbHNlIQ%3D%3D@@1012
###Jacket[path]http://www.thehut.com/catalogue/men/new-in.list?searchFilters=SmFja2V0I0Nsb3RoaW5ndHlwZV9jb250ZW50OkphY2tldCFSlsdGVyI3Jlc3RyaWN0ZWQ6ZmFsc2Uh@@1013
###Knitwear[path]http://www.thehut.com/catalogue/men/new-in.list?searchFilters=S25pdHdlYXIjQ2xvdGhpbmd0eXBlX2NvbnRlbnQ6S25pdHdlYXIhUmVkOmZhbHNlIQ%3D%3D@@1014
![Page 8: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/8.jpg)
Слайд 8 из 22
Скриптыpublic override void RunProduct(RunProductScriptParameters p)
{
product = p.Product;
category = p.Category;
_hpl = (p.Process as OneProductLoader).State.Proxy.GetHtmlPageLoader("http://mail.ru");
product.ID = Helper.GetHashCodeString(product.Name + product.Url);
if (product.Art == "")
product.Art = _HashCod;
if (_imageNameSku)
product.Image = Helper.MakeImgName2Sku(product.ImageFull);
![Page 9: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/9.jpg)
Версия 1
![Page 10: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/10.jpg)
Версия 2
![Page 11: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/11.jpg)
Версия 3
![Page 12: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/12.jpg)
Версия 4
![Page 13: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/13.jpg)
Слайд 13 из 22
VisualWebGuipublic partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void Form1_Load(object sender, EventArgs e)
{
MyAspxBox a = new MyAspxBox();
a.Dock = DockStyle.Fill;
a.Path = "WebForm2.aspx";
this.Controls.Add(a);
}
}
![Page 14: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/14.jpg)
Слайд 14 из 22
VisualWebGui-2public class MyAspxBox : AspPageBox
{
protected override void FireEvent(IEvent objEvent)
{
if (objEvent.Type == "message")
{
string s = objEvent["messagetext"];
MessageBox.Show(s);
}
else
{
base.FireEvent(objEvent);
}
}
}
![Page 15: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/15.jpg)
Слайд 15 из 22
VisualWebGui-3function SendMessage() {
var strMessage = document.getElementById("TextBox1").value;
var objEvent = VWG.Events.CreateEvent("<%= PageContext.Guid %>", "message");
VWG.Events.SetEventAttribute(objEvent, "messagetext", strMessage);
VWG.Events.RaiseEvents();
}
![Page 16: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/16.jpg)
Слайд 16 из 22
Ext.net<script runat="server">
protected void Button_Click(object sender, DirectEventArgs e)
{
X.Msg.Alert("Server Time", DateTime.Now.ToLongTimeString()).Show();
}
</script>
![Page 17: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/17.jpg)
Слайд 17 из 22
Ext.net-2<ext:ResourceManager runat="server"
DirectEventUrl="default.aspx" />
<h1>ImageButton Control Variations</h1>
<h2>Simple ImageButton</h2>
<ext:ImageButton
runat="server"
ImageUrl="button.gif"
OverImageUrl="overButton.gif"
DisabledImageUrl="disabled.gif"
PressedImageUrl="pressed.gif">
<DirectEvents>
<Click OnEvent="Button_Click" />
</DirectEvents>
</ext:ImageButton>
![Page 18: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/18.jpg)
Слайд 18 из 22
Особенности- тестирование
- кэширование
- работа с БД
- задачи + лог
![Page 19: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/19.jpg)
Слайд 19 из 22
Кэширование- предсказуемая скорость
- управляемость
![Page 20: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/20.jpg)
Слайд 20 из 22
Задачи + лог
public class Runnablepublic class TaskInfoзапуск из CMD
![Page 21: CatalogLoader 2014 Belarus Minsk DNUG](https://reader033.fdocument.pub/reader033/viewer/2022042614/557f1d0cd8b42a01678b520f/html5/thumbnails/21.jpg)
Слайд 21 из 22
Работа с БД
- MSSQL, SQL Compact, MySQL
- Http tunnel