opensource

Showing 7 posts tagged opensource

Musings from our CI/CD Meetup: Using Screwdriver, Achieving a Serverless Experience While Scaling with Kubernetes or Amazon ECS, and Data Agility for Stateful Workloads in Kubernetes

By Jithin Emmanuel, Sr. Software Dev Manager, Verizon Media

On Tuesday, December 4th, I joined speakers from Spotinst, Nirmata, CloudYuga, and MayaData, at the Microservices and Cloud Native Apps Meetup in Sunnyvale.

We shared how Screwdriver is used for CI/CD at Verizon Media. Created by Yahoo and open-sourced in 2016, Screwdriver is a build platform designed for continuous delivery at scale.

Screwdriver supports an expanding list of source code services, execution engines, and databases since it is not tied to any specific compute platform. Moreover, it has a fully documented API and growing open source community base.

The meetup also featured very interesting CI/CD presentations including these:

  • A Quick Overview of Intro to Kubernetes Course, by Neependra Khare, Founder, CloudYuga

Neependra discussed his online course which includes some of Kubernetes’ basic concepts, architecture, the problems it solves, and the model that it uses to handle containerized deployments and scaling. Additionally, CloudYuga provides training in Docker, Kubernetes, Mesos Marathon, Container Security, GO Language, Advanced Linux Administration, and more.

  • Achieving a Serverless Experience While Scaling with Kubernetes or Amazon ECS, by Amiram Shachar, CEO & Founder, Spotinst

Amiram discussed two important concepts of Kubernetes: Headroom and 2 Levels Scaling. Amiram also reviewed the different Kubernetes deployment tools, including Kubernetes Operations (Kops). Ritesh Patel, Founder and VP Products at Nirmata, demoed Spotinst and Nirmata. Nirmata provides a complete solution for Kubernetes deployment and management for cloud-based app containerization. Spotinst is workload automation software that’s focused on helping enterprises save time and costs on their cloud compute infrastructure. 

  • Data Agility for Stateful Workloads in Kubernetes, by Murat Karslioglu, VP Products, MayaData

MayaData is focused on freeing DevOps and Kubernetes from storage constraints with OpenEBS. Murat discussed accelerating CI/CD Pipelines and DevOps, using chaos engineering and containerized storage. Murat also explored some of the open source tools available from MayaData and introduced the MayaData Agility Platform (MDAP). Murat’s presentation ended with a live demo of OpenEBS and Litmus.

To learn about future meetups, follow us on Twitter at @YDN or on LinkedIn.

Dash Open Podcast: Episode 02 - Building Community and Mentorship around Hackdays

By Ashley Wolf, Open Source Program Manager, Verizon Media

The second installment of Dash Open is ready for you to tune in!

In this episode, Gil Yehuda, Sr. Director of Open Source at Verizon Media, interviews Dav Glass, Distinguished Architect of IaaS and Node.js at Verizon Media. Dav discusses how open source inspired him to start HackSI, a Hack Day for all ages, as well as robotics mentorship programs for the Southern Illinois engineering community.

Listen now on iTunes or SoundCloud.

Dash Open is your place for interesting conversations about open source and other technologies, from the open source program office at Verizon Media. Verizon Media is the home of many leading brands including Yahoo, Aol, Tumblr, TechCrunch, and many more.

Follow us on Twitter @YDN and on LinkedIn.

Announcing OpenTSDB 2.4.0: Rollup and Pre-Aggregation Storage, Histograms, Sketches, and More

By Chris Larsen, Architect

image

OpenTSDB is one of the first dedicated open source time series databases built on top of Apache HBase and the Hadoop Distributed File System. Today, we are proud to share that version 2.4.0 is now available and has many new features developed in-house and with contributions from the open source community. This release would not have been possible without support from our monitoring team, the Hadoop and HBase developers, as well as contributors from other companies like Salesforce, Alibaba, JD.com, Arista and more. Thank you to everyone who contributed to this release!

A few of the exciting new features include:

Rollup and Pre-Aggregation Storage

As time series data grows, storing the original measurements becomes expensive. Particularly in the case of monitoring workflows, users rarely care about last years’ high fidelity data. It’s more efficient to store lower resolution “rollups” for longer periods, discarding the original high-resolution data. OpenTSDB now supports storing and querying such data so that the raw data can expire from HBase or Bigtable, and the rollups can stick around longer. Querying for long time ranges will read from the lower resolution data, fetching fewer data points and speeding up queries.

Likewise, when a user wants to query tens of thousands of time series grouped by, for example, data centers, the TSD will have to fetch and process a significant amount of data, making queries painfully slow. To improve query speed, pre-aggregated data can be stored and queried to fetch much less data at query time, while still retaining the raw data. We have an Apache Storm pipeline that computes these rollups and pre-aggregates, and we intend to open source that code in 2019. For more details, please visit http://opentsdb.net/docs/build/html/user_guide/rollups.html.

Histograms and Sketches

When monitoring or performing data analysis, users often like to explore percentiles of their measurements, such as the 99.9th percentile of website request latency to detect issues and determine what consumers are experiencing. Popular metrics collection libraries will happily report percentiles for the data they collect. Yet while querying for the original percentile data for a single time series is useful, trying to query and combine the data from multiple series is mathematically incorrect, leading to errant observations and problems. For example, if you want the 99.9th percentile of latency in a particular region, you can’t just sum or recompute the 99.9th of the 99.9th percentile.

To solve this issue, we needed a complex data structure that can be combined to calculate an accurate percentile. One such structure that has existed for a long time is the bucketed histogram, where measurements are sliced into value ranges and each range maintains a count of measurements that fall into that bucket. These buckets can be sized based on the required accuracy and the counts from multiple sources (sharing the same bucket ranges) combined to compute an accurate percentile.

Bucketed histograms can be expensive to store for highly accurate data, as many buckets and counts are required. Additionally, many measurements don’t have to be perfectly accurate but they should be precise. Thus another class of algorithms could be used to approximate the data via sampling and provide highly precise data with a fixed interval. Data scientists at Yahoo (now part of Oath) implemented a great Java library called Data Sketches that implements the Stochastic Streaming Algorithms to reduce the amount of data stored for high-throughput services. Sketches have been a huge help for the OLAP storage system Druid (also sponsored by Oath) and Bullet, Oath’s open source real-time data query engine.

The latest TSDB version supports bucketed histograms, Data Sketches, and T-Digests.

Some additional features include:

  • HBase Date Tiered Compaction support to improve storage efficiency.
  • A new authentication plugin interface to support enterprise use cases.
  • An interface to support fetching data directly from Bigtable or HBase rows using a search index such as ElasticSearch. This improves queries for small subsets of high cardinality data and we’re working on open sourcing our code for the ES schema.
  • Greater UID cache controls and an optional LRU implementation to reduce the amount of JVM heap allocated to UID to string mappings.
  • Configurable query size and time limits to avoid OOMing a JVM with large queries.

Try the releases on GitHub and let us know of any issues you run into by posting on GitHub issues or the OpenTSDB Forum. Your feedback is appreciated!

OpenTSDB 3.0

Additionally, we’ve started on 3.0, which is a rewrite that will support a slew of new features including:

  • Querying and analyzing data from the plethora of new time series stores.
  • A fully configurable query graph that allows for complex queries OpenTSDB 1x and 2x couldn’t support.
  • Streaming results to improve the user experience and avoid overwhelming a single query node.
  • Advanced analytics including support for time series forecasting with Yahoo’s EGADs library.

Please join us in testing out the current 3.0 code, reporting bugs, and adding features.

Join us at the Machine Learning Meetup hosted by Zillow in Seattle on November 29th

By Kristian Aune, Tech Product Manager, Oath

If you are in Seattle on Thursday November 29, please join Jon Bratseth (Distinguished Architect, Oath) at a machine learning meetup hosted by Zillow. Jon will share a Vespa overview and answer any questions about it. Largely developed by Yahoo engineers, Vespa is a big data processing and serving engine, available as open source on GitHub. It’s used by many products, such as Yahoo News, Yahoo Sports, Yahoo Finance, and Oath Ads Platforms.

Eric Ringger, director of machine learning for personalization for Zillow will discuss some of the models used to help users find homes, including collaborative filtering, a content-based model, and deep learning.

Learn more and RSVP here. If you can’t join, please check out these slides from a recent Vespa presentation to learn more about this technology.


The Vespa Team

Oath’s VP of AI invites you to learn how to build a Terabyte Scale Machine Learning Application at TDA Conference

By Ganesh Harinath, VP Engineering, AI Platform & Applications, Oath

If you’re attending the upcoming Telco Data Analytics and AI Conference in San Francisco, make sure to join my keynote talk. I’ll be presenting “Building a Terabyte Scale Machine Learning Application” on November 28th at 10:10 am PST. You’ll learn about how Oath builds AI platforms at scale.

My presentation will focus on our approach and experience at Oath in architecting and using frameworks to build machine learning models at terabyte scale, near real-time. I’ll also highlight Trapezium, an open source framework based on Spark, developed by Oath’s Big Data and Artificial Intelligence (BDAI) team.

I hope to catch you at the conference. If you would like to connect, reach out to me. If you’re unable to attend the conference and are curious about the topics shared in my presentation, follow @YDN on Twitter and we’ll share highlights during and after the event. 
Track

Dash Open 01: Gil Yehuda - Starting, Running, and Improving an Open Source Program Office

Introducing the Dash Open Podcast, sponsored by Yahoo Developer Network

By Ashley Wolf, Principal Technical Program Manager, Oath

Is open source the wave of the future, or has it seen its best days already? Which Big Data and AI trends should you be aware of and why? What is 5G and how will it impact the apps you enjoy using? You’ve got questions and we know smart people; together we’ll get answers. Introducing the Dash Open podcast, sponsored by the Yahoo Developer Network and produced by the Open Source team at Oath.

Dash Open will share interesting conversations about tech and the people who spend their day working in tech. We’ll look at the state of technology through the lens of open source; keeping you up-to-date on the trends we’re seeing across the internet. Why Dash Open? Because it’s like a command line argument reminding the command to be open. What can you expect from Dash Open? Interviews with interesting people, occasional witty banter, and a catchy theme song.

In the first episode, Rosalie Bartlett, Open Source community manager at Oath, interviews Gil Yehuda, Senior Director of Open Source at Oath. Tune in to hear one skeptic’s journey from resisting the open source movement to heading one of the more prolific Open Source Program Offices (OSPO). Gil highlights the benefits of open source to companies and provides actionable advice on how technology companies can start or improve their OSPO.

Give Dash Open a listen and tell us what topics you’d like to hear next.

–

Ashley Wolf manages the Open Source Program at Oath/Verizon Media Group.  

Open-Sourcing Panoptes, Oath’s distributed network telemetry collector

By Ian Flint, Network Automation Architect and Varun Varma, Senior Principal Engineer

The Oath network automation team is proud to announce that we are open-sourcing Panoptes, a distributed system for collecting, enriching and distributing network telemetry.  

We developed Panoptes to address several issues inherent in legacy polling systems, including overpolling due to multiple point solutions for metrics, a lack of data normalization, consistent data enrichment and integration with infrastructure discovery systems.  

Panoptes is a pluggable, distributed, high-performance data collection system which supports multiple polling formats, including SNMP and vendor-specific APIs. It is also extensible to support emerging streaming telemetry standards including gNMI.

Architecture

The following block diagram shows the major components of Panoptes:

image

Panoptes is written primarily in Python, and leverages multiple open-source technologies to provide the most value for the least development effort. At the center of Panoptes is a metrics bus implemented on Kafka. All data plane transactions flow across this bus; discovery publishes devices to the bus, polling publishes metrics to the bus, and numerous clients read the data off of the bus for additional processing and forwarding. This architecture enables easy data distribution and integration with other systems. For example, in preparing for open-source, we identified a need for a generally available time series datastore. We developed, tested and released a plugin to push metrics into InfluxDB in under a week. This flexibility allows Panoptes to evolve with industry standards.

Check scheduling is accomplished using Celery, a horizontally scalable, open-source scheduler utilizing a Redis data store. Celery’s scalable nature combined with Panoptes’ distributed nature yields excellent scalability. Across Oath, Panoptes currently runs hundreds of thousands of checks per second, and the infrastructure has been tested to more than one million checks per second.

Panoptes ships with a simple, CSV-based discovery system. Integrating Panoptes with a CMDB is as simple as writing an adapter to emit a CSV, and importing that CSV into Panoptes. From there, Panoptes will manage the task of scheduling polling for the desired devices. Users can also develop custom discovery plugins to integrate with their CMDB and other device inventory data sources.

Finally, any metrics gathering system needs a place to send the metrics. Panoptes’ initial release includes an integration with InfluxDB, an industry-standard time series store. Combined with Grafana and the InfluxData ecosystem, this gives teams the ability to quickly set up a fully-featured monitoring environment.

Deployment at Oath

At Oath, we anticipate significant benefits from building Panoptes. We will consolidate four siloed polling solutions into one, reducing overpolling and the associated risk of service interruption. As vendors move toward streaming telemetry, Panoptes’ flexible architecture will minimize the effort required to adopt these new protocols.

There is another, less obvious benefit to a system like Panoptes. As is the case with most large enterprises, a massive ecosystem of downstream applications has evolved around our existing polling solutions. Panoptes allows us to continue to populate legacy datastores without continuing to run the polling layers of those systems. This is because Panoptes’ data bus enables multiple metrics consumers, so we can send metrics to both current and legacy datastores.

At Oath, we have deployed Panoptes in a tiered, federated model. We install the software in each of our major data centers and proxy checks out to smaller installations such as edge sites.  All metrics are polled from an instance close to the devices, and metrics are forwarded to a centralized time series datastore. We have also developed numerous custom applications on the platform, including a load balancer monitor, a BGP session monitor, and a topology discovery application. The availability of a flexible, extensible platform has greatly reduced the cost of producing robust network data systems.

Easy Setup

Panoptes’ open-source release is packaged for easy deployment into any Linux-based environment. Deployment is straightforward, so you can have a working system up in hours, not days.

We are excited to share our internal polling solution and welcome engineers to contribute to the codebase, including contributing device adapters, metrics forwarders, discovery plugins, and any other relevant data consumers.  

Panoptes is available at https://github.com/yahoo/panoptes, and you can connect with our team at [email protected].