neural-search

We have separate CI and CD pipelines, which share a lot of code actually (like the testing part). This is a bit annoying to maintain at the moment, because we have to do all changes twice. It would be great do modularize this somehow so that both workflows

A web crawler was added by #775, but the test cases are missing.

As discussed in various meetings with @lusloher , @aga11313 , @FionnD

Kaggle is a lot of hoops for a user to jump through just to get an example working: install, set up key, run data getter script.

It's also work for us: We have to ensure datasets haven't moved or changed a lot, and we sometimes have to perform extra steps to process them.

These datasets are generally under creative com

Current docker image size is insane. It is 2.55 GB. Reduce that to below 1GB or less. Apply changes from this reference: https://hackernoon.com/tips-to-reduce-docker-image-sizes-876095da3b34

Jina includes a Monitor tab in the Console. The Monitor tab connects to a Jina Flow somewhere and displays all the Logs produced by that Flow in a graphical fashion.

Currently, the user can only stream logs from a Flow started by the Console. This happens in the Monitor tab of the Console.
![image](https://user-images.githubusercontent.com/23415764/123066358-46f5a100-d410-1

Intro for core is a scary big wall of text with dozens and dozens of links. This gives the impression there many many pages of content.

The intro is one of the first things new users see. We don't want to terrify them!

Suggestion

One heading and introductory paragraph each for:

• 101
• 102
• Hello world

With link to the full page below. No links to sub pages (and certainly not