Repository navigation
Expand file tree
/
Copy pathindex.html
More file actions
executable file
·1343 lines (1197 loc) · 68.6 KB
/
Copy pathindex.html
File metadata and controls
executable file
·1343 lines (1197 loc) · 68.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en-US" prefix="og: http://opg.me/ns#">
<head>
<meta charset="UTF-8" />
<meta name="title" property="og:title" content="Mashr" />
<meta name="description" property="og:description" content="Mashr is an
easy-to-use data pipeline orchestration and monitoring framework for small
applications." />
<meta name="type" property="og:type" content="website" />
<meta name="url" property="og:url" content="https://mashr-framework.github.io/" />
<meta name="image" property="og:image" content="assets/images/mashr_logo.png" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="author" content="Mat Sachs, Linus Phan, Jacob Coker-Dukowitz" />
<!-- START: favicons -->
<link rel="apple-touch-icon" sizes="57x57" href="favicons/apple-icon-57x57.png">
<link rel="apple-touch-icon" sizes="60x60" href="favicons/apple-icon-60x60.png">
<link rel="apple-touch-icon" sizes="72x72" href="favicons/apple-icon-72x72.png">
<link rel="apple-touch-icon" sizes="76x76" href="favicons/apple-icon-76x76.png">
<link rel="apple-touch-icon" sizes="114x114" href="favicons/apple-icon-114x114.png">
<link rel="apple-touch-icon" sizes="120x120" href="favicons/apple-icon-120x120.png">
<link rel="apple-touch-icon" sizes="144x144" href="favicons/apple-icon-144x144.png">
<link rel="apple-touch-icon" sizes="152x152" href="favicons/apple-icon-152x152.png">
<link rel="apple-touch-icon" sizes="180x180" href="favicons/apple-icon-180x180.png">
<link rel="icon" type="image/png" sizes="192x192" href="favicons/android-icon-192x192.png">
<link rel="icon" type="image/png" sizes="32x32" href="favicons/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="96x96" href="favicons/favicon-96x96.png">
<link rel="icon" type="image/png" sizes="16x16" href="favicons/favicon-16x16.png">
<!-- <link rel="manifest" href="/favicons/manifest.json"> -->
<meta name="msapplication-TileColor" content="#ffffff">
<meta name="msapplication-TileImage" content="favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
<!-- END: favicons -->
<title>Mashr</title>
<!-- <link rel="icon" type="image/png" sizes="32x32" href="assets/logo/mashr_logo_mark.png" /> -->
<link href="https://fonts.googleapis.com/css?family=Open+Sans:800,800i|Raleway:200,400,700" rel="stylesheet" />
<!-- <style>reset</style> -->
<link rel="stylesheet" href="reset.css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.13.1/styles/gruvbox-dark.min.css" charset="utf-8" />
<!-- <style></style> -->
<link rel="stylesheet" href="main.css" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.13.1/highlight.min.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<!-- <script></script> -->
<script src="application.js"></script>
</head>
<body>
<div class="logo-links">
<!-- TODO: change out IDs -->
<img src="assets/logo/mashr_logo_mark.png" alt="Mashr logo" id="mashr-logo" />
<a href="https://github.com/mashr-framework/mashr" target="_blank">
<img src="assets/icons/github_white.png" alt="github logo" id="github-logo" />
</a>
</div>
<nav id="site-navigation">
<ul>
<li>
<a href="#home" id="home-link">HOME</a>
</li>
<li>
<a href="#case-study" id="case-study-link">CASE STUDY</a>
<nav id="case-study-mobile">
<ul></ul>
</nav>
</li>
<li>
<a href="#our-team" id="our-team-link">OUR TEAM</a>
</li>
</ul>
</nav>
<header id=home>
<h1>
<img src="assets/logo/mashr_logo.png" alt="Mashr logo"/>
</h1>
<p>a <span>data pipeline</span> framework built for the cloud</p>
</header>
<section class="integration">
<article class="box">
<div class="text-box">
<h1>Automate your Data Pipeline</h1>
<p>In today’s world even small applications use a variety of
tools to get the job done. Mashr simplifies the process of
getting data from all of those tools into a single place so that
you can use it.</p>
<!-- <a class="button" href="#case-study">Learn More</a> -->
</div>
</article>
<div class="box">
<img src="assets/images/integration.png" alt="integrate your data" />
</div>
</section>
<section class="integration">
<article class="box">
<div class="text-box">
<h1>Built with Best Practices</h1>
<p>Mashr is optimized for data pipeline best practices including
monitoring, archiving, and backup failover strategy in case of
failure.</p>
<!-- <a class="button" href="#case-study">Learn More</a> -->
</div>
</article>
<div class="box">
<img src="assets/images/best_practices.png" alt="best practices" />
</div>
</section>
<main>
<section id="case-study">
<h1>Case Study</h1>
<nav>
<ul></ul>
</nav>
<h2 id="introduction">1 Introduction</h2>
<h3>1.1 What is Mashr?</h3>
<p>Mashr is an open source easy-to-use data pipeline orchestration and monitoring
framework for small applications. Mashr simplifies the process of taking
your data from disparate sources and putting them into a single place so
that you can use that data. It is optimized for data pipeline best
practices including monitoring for each step of your data pipeline along
with archiving and backup in case of failover. Mashr is built with
Node.js, provides an easy-to-use CLI, and uses Docker to host Embulk on
GCE Instances on Google Cloud Platform (GCP).</p>
<p>In today’s world even small applications use a variety of tools to
get the job done. It’s reasonable to assume that a small application
would have data in over a dozen sources including email applications,
ecommerce platforms, and more. Without a data pipeline that transfers
data from multiple sources to a central repository, it can be difficult
to perform analysis and make sound business decisions.</p>
<p>Cloud platforms give the promise of easy-to-use solutions to answer
problems like this. The irony, however, is that cloud platforms can
often still be cumbersome to use and offer an overwhelming array of
choices that can make life difficult for a developer. Mashr was built to
simplify the process of deploying data pipelines in the cloud so that
developers can focus on building their application and not the data
pipeline.</p>
<p>This case study will begin by describing some of the best practices
for building a data pipeline, as well as the challenges that developers
face in building their own. Next we’ll talk about some of the existing
solutions that solve this problem, and why we built Mashr to address
some of the drawbacks of those solutions. Finally, we’ll discuss the
design questions we made while building Mashr and some of the unique
challenges that came up along the way.</p>
<h3>1.2 Why GCP?</h3>
<p>We decided to implement the Mashr data pipeline on GCP first,
because of it's robust set of tools for data analysis. When comparing
the tools for big data analytics between Google’s GCP and Amazon’s AWS,
GCP has an edge due to their varied range of services, including
Tensorflow which is one of the most used Machine Learning libraries
<sup><a href="#footnote-3">3</a></sup>.</p>
<h2 id="data-pipelines">2 Data Pipelines</h2>
<h3>2.1 Use Case</h3>
<p>In order to understand why we need data pipelines in the first place,
let’s consider a scenario. Imagine a small web application called
“Bookend” that sells used books to a niche market. Bookend has sales
data from their ecommerce platform, sales data from Amazon, and email
data from MailChimp.</p>
<p>Bookend wants to know how many emails were opened and how “opens”
affected sales on their two sales platforms. This means that they need to
get all of these different data sources together into a single
destination to perform analysis on it. In addition, Bookend doesn’t just
want to do this once. They need to do it every day so that they can
understand how well their email marketing is working over time. That
means that they need an automated solution that can take data from these
different sources and combine them into a single destination.</p>
<p>At first, it may look like their use case is simple and they could just
build a small application that makes API calls regularly to get data
into a single destination database where they can do reporting:</p>
<img src="assets/images/use_case_a.png" alt="use case A" />
<p>Unfortunately, they need a little more than this simple diagram:</p>
<ul>
<li>
If a job fails to load to the destination database, they want to
be able to keep a record of it so that they can see what went
wrong and so that they don’t lose any data.
</li>
<li>
They need to keep an archive of successful load jobs to the
database as well, so that they can debug issues if any data is
corrupted.
</li>
<li>
They will be performing some transformation and validation on the
data before it is loaded into the destination database.
</li>
</ul>
<p>Lastly, Bookend is a small but growing application and they will
want to add new sources of data in the future. For instance, they might
begin selling on another ecommerce platform like eBay, or use a
different mail provider or add a text messaging system. Therefore they
would like an easy way to add new integrations as needed without a lot
of overhead.</p>
<p>After considering those features, the diagram of what they want looks
more like this:</p>
<img src="assets/images/use_case_b.png" alt="use case B" />
<p>Data pipeline tools and services exist to address just this kind of
scenario. As you can see, if Bookend started their journey knowing only
that they needed a data pipeline, they would quickly be confronted with
a number of design decisions. This is why there is a large ecosystem of
tools and platforms that perform data pipeline services. The range of
these tools is vast. For instance, you have everything from Apache
Airflow <sup><a href="#footnote-7">7</a></sup> which is an open source
workflow management system, to Stitch <sup><a href="#footnote-8">8</a></sup>,
a proprietary solution with an easy-to-use drag-and-drop UI.</p>
<p>In order to better understand why there is such a large array of
tools out there and what design decisions Bookend would face, let’s
consider some guiding principles for building a robust data pipeline
for Bookend.</p>
<h3>2.2 Principles of a Data Pipeline</h3>
<h4>2.2.1 Extract Data from the Source</h4>
<p>While it may sound self-evident, we should first think about how we
pull data from the source. Data may come in many different formats
including XML, JSON, CSV, etc. Your application will need to change the
data into a format that the destination database can read. To do these
things the application will have to live somewhere, so you will have to
consider where you host it. You’ll also have to consider how you schedule
the extraction of data at a regular interval. For instance, you could
schedule extraction from a cron job or another tool.</p>
<h4>2.2.2 Validate</h4>
<p>Oftentimes we will want to perform some kind of validation on the data
before it reaches our destination database. Specifically, we would want
to ensure that the data was not corrupted in the source, corrupted by
the data pipeline itself, and that no data was dropped in the previous
steps before loading the data into the destination.</p>
<p>For instance, imagine that one of the sources of data was a database
with financial transactions from the past year. We may want to make sure
that there is a date field and that this field has valid dates within
the past 12 months.</p>
<p>Validation can be an important concern because for many destination
databases, if you try to load data that is different than what the
destination expects, the load job may fail.</p>
<h4>2.2.3 Transform</h4>
<p>More often than not we will want to perform some sort of transformation
on the data <sup><a href="#footnote-1">1</a></sup>. This could look like
a lot of different things. For example:</p>
<ul>
<li>Removing extraneous or erroneous data (cleaning)</li>
<li>Encoding free-form values</li>
<li>Translating coded values</li>
<li>Deriving a new calculated value</li>
<li>Splitting a column into multiple columns</li>
</ul>
<p>For example, we would want to encode free-form values if the source
has a field “male” that we want to map to “M” in the destination. We
would want to translate coded values if the source codes female as “1”
and the destination codes female as “F”. There are many more
transformations that you may need to do before data reaches the
destination database.</p>
<h4>2.2.4 Staging / Archiving</h4>
<p>As we mentioned above, the load job will fail with many databases if
you try to load data in the wrong format. You will need to consider
what to do with your data if the data fails to load to the destination
database. One solution is to not load data directly into the
destination database. Instead, the data should first enter a staging
area. We then attempt to perform a load job from this staging location.
The data could then be archived if the load job is successful, and held
in the staging area if the load job fails. If we keep an archive of the
data, our data pipeline can be more resilient to failover and we can
make it easier to rollback if something goes wrong. In addition to
handling errors, we can use the archive and staging locations to
generate audit reports or diagnose and repair problems.</p>
<h4>2.2.5 Monitoring</h4>
<p>Because load jobs are not always successful and data can become
corrupted, we need an easy way to debug each step of our data pipeline.
Therefore, we should have a logging service that takes account of:</p>
<ol class="lower-alpha-list">
<li>What action was taken</li>
<li>If there were errors that occurred</li>
<li>What data was transferred in each step of our data pipeline</li>
</ol>
<p>Ideally, if you were to see data corrupted in your destination
database then you should have a method to review every step of you data
pipeline to see where the corruption occurred.</p>
<img src="assets/images/monitoring.png" alt="Monitoring"/>
<h4>2.2.6 Loading Data</h4>
<p>Another area that appears self-evident but in fact has hidden
complexity is the actual method by which you load data into the
destination. For example, some systems like fraud detection may require
data to be handled in real-time or near real-time. For this reason,
many cloud providers offer a “streaming” method to load data into their
databases in addition to a “batch” loading method. There are tradeoffs
for either including financial cost and setup that you may need to
consider.</p>
<h4>2.2.7 Orchestration and Ordering of tasks</h4>
<p>A common source of problems in a data pipeline is that there can be a
large number of dependent tasks. In this example, “B” should not start
without “A” completing, then “C” and “D” depend on a condition in “B”.
A data pipeline should be able to have a simple way to ensure that these
kinds of dependencies are easy to organize, and has mechanisms to ensure
the successful completion of complex tasks.</p>
<img src="assets/images/b_depends_on_a.png" alt="Orchestration and Ordering of tasks" />
<h4>2.2.8 Performance</h4>
<p>Finally, a data pipeline should consider constraints on performance
and we should make any optimizations where possible. This is especially
true when dealing with a large volume or frequency of data transfer.</p>
<p>For example, many data pipelines may make use of parallel processing.
A simple example of this would be taking one large file and splitting
it up into many smaller files that are processed at the same time.</p>
<p>You should also examine your data pipeline for bottlenecks. These
frequently occur in two areas: </p>
<ul>
<li>
Pulling data from the source: a source may be an API that throttles
too many requests or a database with read limits.
</li>
<li>
Loading data to the destination database: usually because of write
limits or other constraints specific to the kind of database that
you are loading data into.
</li>
</ul>
<h3>2.3 Flow of Data</h3>
<p>Once you’ve thought through which of these features your data
pipeline will implement and how it will implement them, you’ll then
want to consider in what order your data pipeline flows. There are two
specific subsets of data pipelines that you may hear about.</p>
<h4>2.3.1 ETL</h4>
<p>The first subset is ETL, which stands for “Extract, Transform,
Load”. This follows the traditional flow of data for most data
pipelines. Due to the cost of storing data, you would want to perform
any transformations that reduce the amount of data before loading it to
the destination database <sup><a href="#footnote-6">6</a></sup>. For
instance you may want to perform aggregations on the data or another
step that would reduce the amount of data before loading.</p>
<img src="assets/images/etl.png" alt="Extract, Transform and Load" />
<h4>2.3.1 ELT</h4>
<p>The other subset is ELT, which stands for “Extract, Load, Transform”.
With the advent of cloud platforms like Google Cloud Platform and AWS,
the cost of storing data is much cheaper and therefore is no longer as
great a concern <sup><a href="#footnote-6">6</a></sup>. Therefore,
transformations like aggregation can occur after we store data.</p>
<img src="assets/images/elt.png" alt="Extract, Load and Transform" />
<h3>2.4 Components of a data pipeline</h3>
<p>After reviewing the principles of a good data pipeline for our use
case, we can see that there are four major components of a data pipeline
from an infrastructure perspective:</p>
<img src="assets/images/data_pipeline_components.png" alt="Components of a Data Pipeline"/>
<h3>2.5 Challenges of Building a data pipeline on GCP</h3>
<p>We have already shown that there are actually a lot of principles
and decision points to consider when building a data pipeline solution
that automatically transfers data from one place to another. Let’s
briefly explore what it takes to set up a data pipeline on Google Cloud
Platform (GCP).</p>
<h4>2.5.1 Overwhelming number of configuration options</h4>
<p>Setting up a data pipeline on GCP can be a complicated and
challenging task because of the large number of services that are
offered that may be relevant for setting up a data pipeline. For
example, some of the services that are currently offered are GCP’s Data
Transfer Service, DataFlow, Cloud Functions, Cloud Run, and so much
more. Additionally, new products come out almost every day, and existing
products are changed frequently. It starts to get overwhelming quick.</p>
<p>After deciding which services to use, the developer still has a lot
of work to do before being competent in using those services. As an
example, this table shows a count of how many API calls and GCP services
that Mashr manages in GCP to set up a data pipeline. A developer setting
out to create their own data pipeline would need to be familiar with
each of these services and the configuration options available to each
service. This table also doesn’t mention the countless validations as
well as the setup required for logging and other helpful tools that
Mashr provides.</p>
<img src="assets/images/mashr_api_calls_to_gcp_table.png" alt="Number of API calls to GCP: 23; Number of GCP Services used: 7" />
<h4>2.5.3 Incomplete libraries</h4>
<p>Also, some of the client libraries for communicating with GCP
services may not be complete. For instance, there is no node.js client
library to manage cloud functions at the time of this writing
<sup><a href="#footnote-11">11</a></sup>.</p>
<h4>2.5.4 Individual “tasks” are comprised of many smaller tasks</h4>
<p>Finally, there is rarely a single task that doesn’t involve a series
of smaller tasks. For instance, creating a virtual machine instance on
the cloud with GCP requires you to also determine what OS you need and
how much memory it should have <sup><a href="#footnote-10">10</a></sup>.
As another example, setting up a Google Cloud Storage Bucket requires
you to determine which Storage Class to use
<sup><a href="#footnote-9">9</a></sup>.</p>
<h2 id="solutions">3 Solutions</h2>
<p>Due to the complexity of setting up a data pipeline on your own,
there are many solutions currently in existence that seek to ease the
process.</p>
<h3>3.1 Hosted Solutions</h3>
<p>First, there are “hosted” solutions, which are either fully or
partially proprietary solutions that abstract away server setup and
deploys an entire data pipeline for you.</p>
<p>Most of these services provide a graphical interface and
plug-and-play functionality. They range from enterprise grade services
like Informatica PowerCenter that can cost over $100,000, to tools such
as Stitch and FiveTran which provide affordable tiers to small and
medium sized companies.</p>
<p>There are several good reasons to go with a hosted solution:</p>
<ul>
<li>
Both the infrastructure and the processes for dealing with
extraction and loading is abstracted from the user—there are no
servers to consider.
</li>
<li>Many deal with failover and backup of your data.</li>
<li>
Also, many have a graphical user interface which makes these
solutions convenient and easy to use.
</li>
</ul>
<p>There are also some downsides:</p>
<ul>
<li>
Data sits on someone else’s server which is just one more
vulnerability you have to worry about.
</li>
<li>
There is less control and customization available to you as a
developer.
</li>
</ul>
<img src="assets/images/pros_cons_hosted_solutions.png" alt="pros and cons table for hosted solutions" />
<h3>3.2 Self-Hosted Solutions</h3>
<p>The second group of solutions are "self-hosted" solutions. These are
open source tools which allow you to implement your own data pipeline.
These solutions don’t handle hosting for you and require more
configuration. Unfortunately, many of these solutions are
unmaintained.</p>
<p>As an example, one of the surviving and most successful of these
open source platforms is Pentaho Data Integration (PDI)
<sup><a href="#footnote-12">12</a></sup> which integrates with over
300 services. PDI's graphical interface can be used to develop a
simple ETL data pipeline for extraction, transformation, and
loading.</p>
<p>The pros of self-hosted solutions are:</p>
<ul>
<li>
They are very customizable. You can edit or extend the code as
needed. If an integration doesn’t exist that you need you could
always build one.
</li>
<li>
You don’t need to worry about the data sitting on anyone else’s
servers.
</li>
<li>They are cheaper than fully hosted.</li>
</ul>
<p>The downside to using a self-hosted solution is that their
customizability comes at a cost.</p>
<ul>
<li>
If you want a fully automated solution you will still need to host
it somewhere.
</li>
<li>
You will need to learn the application and set up the
infrastructure for it. For example, you will likely need to worry
about managing the scheduling, archiving, monitoring, or failover
for your data.
</li>
</ul>
<img src="assets/images/pros_cons_self_hosted_solutions.png" alt="pros and cons table for self-hosted solutions" />
<h3>3.3 Mashr</h3>
<p>Recall that Mashr is built for small applications that have data
spread out over a variety of data sources such as a psql database,
salesforce, or other applications with REST API’s. Developers for the
small application want to combine all of that data into one place so
they can use it.</p>
<p>Using a self-hosted solution sounds better since it is customizable,
and already partially implemented. However these solutions usually don’t
account for the hosting, scheduling, monitoring, or failover necessary
in a production environment. They still take a lot of time to set up
hosting and to make all of the orchestration decisions.</p>
<p>This is where Mashr fits in—Mashr’s goal is to provide all the
extraction, transformation, and loading functionality a small
application needs, while being customizable and simple for a developer
to deploy. Mashr does all of this while also taking into account the
best practices for setting up a data pipeline.</p>
<img src="assets/images/abstracted.png" alt="showing where mashr sits on scale of level of abstraction" />
<p>In the image above, the word “abstracted” means how much of the
hosting, management and configuration a user has to worry about.</p>
<p>Mashr sits somewhere in the middle. It is less abstracted than a
hosted solution, and more abstracted than a self-hosted solution. Mashr
makes decisions for the user about the scheduling and running of jobs,
how to recover data that fails to load, or where to archive data for
future reference.</p>
<img src="assets/images/customizable.png" alt="showing where mashr sits on scale of level of customizability" />
<p>Similarly with “customizability”, Mashr is positioned between hosted
and self-hosted solutions. Since Mashr is open source, a user can alter
the code when necessary.</p>
<p>Hosted solutions are closed systems that have limited
customizability. At the same time, since Mashr has more functionality
built-in than many self-hosted options, it’s less customizable compared
to typical self-hosted solutions. Those solutions leave hosting and
orchestration details up to the user to customize themselves.</p>
<p>The pros for using Mashr include:</p>
<ul>
<li>
You don’t have to worry about the data sitting on someone else’s
servers.
</li>
<li>
More abstraction than a self-hosted solution; it manages the set up
of infrastructure for your data pipeline.
</li>
<li>
More customizability than the hosted solutions. You can add new
integrations, and it provides an easy way to add your
transformation and validation steps.
</li>
</ul>
<p>The cons for using Mashr are that:</p>
<ul>
<li>
While Mashr sets up the data pipeline infrastructure, it leaves it
up to the user to manage it. For instance, the user may have to go
into those services and debug them if an error appears.
</li>
<li>
Mashr is opinionated about how a data pipeline should be set up on
GCP. If your use case is more complicated or you want to use other
GCP tools then that would mean editing the actual code base of
Mashr or using GCP’s UI.
</li>
</ul>
<img src="assets/images/pros_cons_mashr.png" alt="pros and cons table for mashr" />
<h2 id="mashr">4 Mashr</h2>
<img src="assets/images/mashr_deploy.gif" alt="gif of running `mashr deploy` in the terminal" />
<p>Mashr was made to be an easy-to-use framework with just a few
commands so that developers can get started quickly building their own
data pipelines. Below are the main commands with a brief description:
</p>
<ul>
<li>
<code class="command">init</code>
- creates a YAML configuration file in the users working
directory
</li>
<li>
<code class="command">deploy</code>
- launches all of the GCP services to create the data pipeline
</li>
<li>
<code class="command">destroy</code>
- destroys all of the GCP services of a specific data pipeline
</li>
<li>
<code class="command">list</code>
- lists your current data pipelines
</li>
<li>
<code class="command">help</code>
- help text for Mashr
</li>
</ul>
<p>The workflow for a developer looks like this:</p>
<ul>
<li>
First, you would run <code class="command">mashr init</code>
which sets up the user’s current working directory as a mashr
directory. It would create a <code>mashr_config.yml</code> file in
the user’s current working directory.
</li>
<li>
The user then fills out the <code>mashr_config.yml</code> file
to tell Mashr what data source it will be pulling from and what
BigQuery dataset and table the data should go to.
</li>
<li>
Then you run <code class="command">mashr deploy</code>, this
launches all of the services needed.
</li>
<li>
Finally, <code class="command">mashr destroy</code> will take down
all those services.
</li>
</ul>
<p>Before going into detail about the data pipeline that Mashr deploys
and what these commands do behind the scenes, it’s important to
understand the tools and GCP services that Mashr uses.</p>
<h3>GCP services used</h3>
<img src="assets/images/gcp_services.png" alt="GCP services that Mashr uses" />
<p>When Mashr deploys to GCP it makes use of a number of services,
shown in the table above. Readers may already be familiar with AWS’s
cloud services, so we’ll name the equivalent service on AWS to help
build a frame of reference.</p>
<h5>Google Compute Engine (GCE)</h5>
<p>GCE is the Infrastructure-as-a-Service (IaaS) component of GCP. A
GCE Instance is a virtual machine. If you’re familiar with AWS this is
similar to an EC2 instance.</p>
<h5>Google Cloud Storage (GCS)</h5>
<p>GCS is a data storage service similar to AWS’s S3. GCS users create
“buckets” to store data. Each bucket can have different settings that
affect the speed and price of the storage. For example Mashr uses both
“Multi-Regional” and “Coldline” storage. Multi-Regional is faster but
more expensive than Coldline storage.</p>
<h5>Cloud Function</h5>
<p>Cloud Functions are GCP’s implementation of a Function-as-a-Service
(FaaS) platform, similar to AWS Lambda. It executes custom code on
fully-managed servers in response to event triggers. Since the servers
are fully managed, this removes the work needed to manage the
infrastructure that the code runs on.</p>
<h5>BigQuery</h5>
<p>BigQuery is GCP’s data warehouse, similar to AWS’s Redshift.
BigQuery enables interactive analysis of massively large datasets. It
is an analytical engine that provides online analytical processing
(OLAP) performance.</p>
<h5>Stackdriver</h5>
<p>Stackdriver is GCP’s logging service, similar to AWS’s CloudTrail or
CloudWatch. Stackdriver aggregates metrics, logs, and events from cloud
and on-premises infrastructure. Developers can use this data to quickly
pinpoint causes of issues across a system of services on GCP.</p>
<h3>4.2 Embulk</h3>
<p>One key component in a robust data pipeline is the data extraction
component, which is the component that's responsible for managing the
scheduling and extraction of data from the source. Mashr uses Embulk
<sup><a href="#footnote-13">13</a></sup> as part of its default data
extraction component, and we'll spend this section understanding why
we use Embulk, what it is, and how Mashr uses it.</p>
<h4>4.2.1 Why we use Embulk</h4>
<p>When building Mashr, we first considered how users would supply data
to the pipeline without having to write a lot of code. We wanted
something modular and easy to use. There were several options we looked
at. We decided on Embulk because it has a large open source community
and well maintained code base. </p>
<h4>4.2.2 What is Embulk</h4>
<p>In many ways Embulk fits into the category of ‘self-hosted’ solutions
that we talked about earlier. It is a plugin-based tool that can extract
and load data to and from many sources, but does not manage archiving,
backup, monitoring or scheduling and hosting for the user.</p>
<p>By “plugin-based”, we mean that a user can choose from a variety of
existing plugins for their input and output data sources. For instance,
Embulk features plugins for REST APIs, relational and nosql databases. A
developer can also build their own plugins if there is not an existing
one for their data source.</p>
<p>Users download Embulk as a JRuby application. Then users can download
various gems (dependencies) which are the input and output plugins that
allow users to transfer data from one service to another. Embulk allows
input and output plugins to communicate with each other via Ruby objects
in a standard format so that any input data source can talk to any
output destination. Embulk takes an <code>embulk_config.yml</code> file
that specifies the input and output for a specific Embulk job, and then
that Embulk job can be run in the terminal with the command
<code class="command">embulk run embulk_config.yml</code>.</p>
<h4>4.2.3 How Mashr uses Embulk</h4>
<p>As mentioned above, Mashr uses Embulk as the data extraction
component of the data pipeline it creates on GCP.</p>
<p>Mashr requires that a user completes a <code>mashr_config.yml</code>
file before deploying the data pipeline to GCP. This
<code>mashr_config.yml</code> file specifies several values that Mashr
needs to build the data pipeline on GCP, as well as values that Embulk
will need to extract data from the source.</p>
<p>For instance, the <code>mashr_config.yml</code> file allows the user
to specify what dependencies (plugins) are needed to run Embulk for a
particular job, and what <code>run</code> command will be used when the
Embulk job is run. There are different commands to run an Embulk job
depending on your use case. Two <code class="command">run</code> commands
that are important to know are:</p>
<ul>
<li>
<code class="command">embulk run config.yml</code>
- Runs an embulk job
</li>
<li>
<code class="command">embulk run config.yml -c diff.yml</code>
- Runs an embulk job with incremental loading
</li>
</ul>
<p>An “incremental load job” in Embulk is a job that tracks what data
has been loaded to prevent sending values that have already been loaded
to the destination. To do this, Embulk will create a second YAML file
that keeps track of the last values loaded.</p>
<img src="assets/images/incremental_loading.png" alt="embulk input yaml configuration file with incremental loading" />
<p>In this example embulk yaml file, you can see that the value for
<code>incremental</code> is set to <code>true</code>, and that the
columns that Embulk will track for incremental loading are
<code>created_at</code> and <code>id</code>. When we run this job with
the command <code class="command">embulk run config.yml -c
diff.yml</code>, Embulk will create a separate <code>diff.yml</code>
file that keeps track of the last values of <code>created_at</code> and
<code>id</code> that have been loaded. Embulk will then check those
values the next time <code class="command">embulk run config.yml -c
diff.yml</code> runs.</p>
<p>Finally, Mashr uses the input data source that the user specified in
the <code>mashr_config.yml</code> file to create a new
<code>embulk_config.yml</code> file with the output set to a GCS bucket.
We’ll explain why the GCS bucket is the destination later on. Here’s an
example of the <code>embulk_config.yml</code> file that Mashr
generates:</p>
<img src="assets/images/example_embulk_config_file.png" alt="example embulk_config.yml file" />
<p>We’ll discuss more about how this works on section 4.4.1 when we talk
about the command <code class="command">mashr init</code>.</p>
<h3>4.3 Mashr Architecture</h3>
<p>Now that we have a basic understanding of the GCP services and
tools that Mashr uses, let’s talk about what Mashr does and how it was
built. This diagram shows a high-level overview of Mashr’s main
components. When you run <code class="command">mashr deploy</code> in
your terminal, the following actions take place:</p>
<img src="assets/images/main.png" alt="Mashr architecture" />
<ol class="number-list">
<li>
Mashr sets up a GCE instance, with Embulk running on a Docker
container.
</li>
<li>
The container has a cron job running that pulls data from the
source and loads it into Google Cloud Storage.
</li>
<li>
Adding the data to GCS triggers the Cloud Function. The Cloud
Function attempts to load the data into BigQuery.
</li>
<li>
If the load is successful, the Cloud Function moves the data file
from the staging bucket to the archive bucket.
</li>
<li>
If the load is not successful, the data file remains in the
staging bucket for debugging purposes.
</li>
<li>
Each step of this process has Stackdriver monitoring enabled so
users can debug the data pipeline if necessary.
</li>
</ol>
<p>Let’s dive into the components of Mashr so we can break down each
step described above. We’ll start by talking about Embulk and the GCE
instance it is hosted on.</p>
<h3>4.3.1 Google Compute Engine & Docker</h3>
<p>The data extraction component needs to be hosted somewhere. We
considered using a Cloud Function, but this wouldn’t work for two
reasons. First, Embulk runs on the JVM and Cloud Functions don't
support JVM. Second, we needed to persist data between runs to track
the most recent record extracted, however Cloud Functions are ephemeral
and therefore do not persist data between instantiations runs on their
own.</p>
<img src="assets/images/GCE.png" alt="Google Compute Engine and Embulk
on Docker Container" /> <p>We decided it made sense to use a GCE
(virtual machine) instance to host Embulk, which allows us maximal
flexibility and the ability to persist data between jobs.</p>
<p>Hosting on a GCE Instance raised the concern of OS dependency. For
instance, GCP recently discontinued support for GCE instances running
Debian 8. Over time, as VMs evolve and older versions are dropped from
the GCE library, Mashr may break and need to be reconfigured to run on
new OSs. It’s fine to upgrade over time, but we want to do upgrades on
our terms rather than be forced to update on someone else’s schedule.
For this reason, we put Embulk in a Docker container to make the
application resilient to technology changes. We’ll discuss this process
in more detail in section 5.</p>
<img src="assets/images/docker_image_embulk.png" alt="Docker image for Embulk" />
<p>We built a custom Docker image that creates a container with
several features and customizations depending on each particular
integration’s configuration. For instance, each container will need
different Embulk plugins installed and different Embulk run commands
depending on the user’s needs.</p>
<p>Additionally, we implemented the scheduling of Embulk jobs with a
cron job that runs every 10 minutes in the Docker container. By
default, containers shutdown once the main process running in the
foreground ends. So we needed the container to stay up between runs.
We implemented a solution to keep the container active that we’ll go
into detail about in the ‘challenges’ section below.</p>
<p>Finally, our Docker container is configured to send logs to GCP’s
Stackdriver for easier debugging and monitoring. This proved to be a
little tricky, and we talk about how we overcame this in the
"Challenges" section at the end of this document.</p>
<h3>4.3.2 Google Cloud Storage</h3>
<p>The Embulk job running on the GCE instance formats that data into a
newline-delimited-JSON file, which works well for loading data into
BigQuery. However, instead of loading the data directly into BigQuery
we have an intermediary step where the data is loaded to a staging
bucket in GCS. We use the GCS buckets to provide a way to debug load
issues if they occur, and archive data files in case they’re needed
later. We’ll explain the archiving process in more detail shortly.</p>
<img src="assets/images/GCS.png" alt="Google Cloud Storage staging and archive buckets" />
<h3> 4.3.4 Cloud Function and BigQuery </h3>
<p>When Embulk loads the data file into GCS, the Mashr Cloud Function
is triggered. When the function triggers it first checks if the
destination table named in the configuration file exists in the
BigQuery dataset. If not, Mashr will create the table before attempting
to load data.</p>
<p>The Mashr Cloud Function’s primary purpose is to load the data file
from the GCS staging bucket into BigQuery. After a successful load, the
Function moves the data file from the GCS Staging bucket into the
Archive bucket. The Archive bucket uses Coldline storage so that large
amounts of data can be stored cheaply. We’ll explain the purpose of the
Archive bucket in the next section.</p>
<img src="assets/images/GCF_to_GBQ.png" alt="Google Cloud Function to BigQuery" />
<p>Earlier, during the deployment process on the user’s local machine,
the code for this Cloud Function is generated and stored in the user’s
current working directory. This enables users to make customizations to
it if desired. For example, they could implement customized data
transformations or validations.</p>
<p>Throughout the process, the logs from the Cloud Function are also
sent to Stackdriver so the user can easily debug if necessary.</p>
<h3>4.3.5 Failover for BigQuery Loads</h3>
<p>This brings us to how Mashr handles a failing load job and what the
Archive bucket is for. As previously stated, when the load job is
successful the Cloud Function moves the data file from the Staging to
the Archiving bucket for long term storage. However, if the data fails
to load to BigQuery, the Cloud Function leaves the file in the Staging
bucket. This way a user can easily find data files that were not loaded
into BigQuery and investigate the cause. For example, if a new field
was added to the data, BigQuery will reject the load. Once the user
identifies the issue they can fix the problem and then load the data
manually.</p>
<img src="assets/images/GCS.png" alt="Google Cloud Storage staging and archive buckets" />
<h3>4.4 Mashr Commands</h3>
<p>In order to understand how much work Mashr is doing for the
developer, let's explore what each command is doing behind the scenes,
starting with <code>init</code>.</p>
<h3 id="4.4.1">4.4.1 init</h3>
<p>When you run <code>init</code> a template YAML configuration file
is created in your working directory called
<code>mashr_config.yml</code>. You can see that it instructs the user
to fill out the template before running
<code class="command">mashr deploy</code>:</p>
<img src="assets/images/mashr_init.gif" alt="gif of running `mashr init` in the terminal" />
<p>In this example you can see that we’ve run <code>init</code> and
Mashr created a <code>mashr_config.yml</code> template file in the
user’s current working directory.</p>
<img src="assets/images/mashr_config_a.png" alt="showing mashr_config.yml file after running `mashr deploy`" />
<p>If you pass the <code>--template</code> flag to <code>init</code>,
you can specify some different templates for various input data sources
that Mashr has available, including <code>psql</code> and
<code>http</code>. This example is of an <code>http</code> template.
In the actual templates that are generated, there are helpful comments
in each section that explain what to input and how Mashr will use it.
</p>
<p>The user then only has to fill in the details and run <code class="command">mashr
deploy</code>, and Mashr will do the heavy lifting of setting up your
data pipeline.</p>
<p>This is the example <code>mashr_config.yml</code> file after the
user has filled it out. The first section at the top,
<code>mashr</code>, is where the user specifies their GCP credentials
along with the table and dataset to send data to. Additionally, the
user specifies an <code>integration_name</code>. The
<code>integration_name</code> is what Mashr uses to name the GCS
buckets, the GCE Instance and the Cloud Function. This way, the user
can find them easily in GCP’s console if needed.</p>
<img src="assets/images/mashr_config_b.png" alt="showing example mashr_config.yml file that is filled out" />
<p>Also in the <code>mashr</code> section, the user can specify what
command to run Embulk with, and any dependencies or gems that this
integration needs to run. These will be used later when we set up the
Docker container with Embulk running on it inside the GCE instance.</p>
<p>Everything inside the <code>embulk</code> section specifies what
the input should be. You can see that the user has specified an input
type of <code>http</code> and the url to <code>get</code> from.</p>
<h3>4.4.2 Deploy</h3>
<p>After filling out the <code>mashr_config.yml</code> file, the user
runs <code class="command">mashr deploy</code> from the same directory,
and Mashr will launch all the services necessary in GCP to create the
data pipeline.</p>
<img src="assets/images/mashr_deploy.gif" alt="gif of running `mashr deploy` in the terminal" />
<p>Deploy does a lot of work for you in the background. As a broad
overview, deploy performs these steps:</p>
<ol class="number-list">
<li>
Performs local validations to ensure that the
<code>integration_name</code>, table name and dataset name fit
GCP’s naming conventions.
</li>
<li>
Performs remote validations with GCP to ensure that the names for
the Cloud Function, GCE Instance and GCS Buckets are not in use. As
an example, GCS bucket names need to be globally unique across all
of GCP.
</li>
<li>
Copies a cloud function template to the current working directory
so that the user can edit it if they wanted to add validations or
transformations inside their data pipeline.
</li>
<li>
Generates various files needed for the Embulk container including
the crontab, the installation script for gems that our Embulk
instance will need, and the Embulk yaml file used to extract data
from the source.
</li>
<li>
Asynchronously launches all of the GCP services we talked about
earlier: the GCE Instance, the GCS Buckets, the Cloud Function, and
the BigQuery dataset and table.
</li>
<li>
Finally, once the GCE Instance is created, the docker image is
built and the docker container is started using the files generated
in step 4.
</li>
</ol>
<p>Here is a diagram showing some of those major steps.</p>
<p>
<img src="assets/images/mashr_deploy_flow_chart.png" alt="flow chart showing some of the major steps of running `mashr deploy`" />
</p>