AWS Lambda part 3 – IO with Files using S3

Previously I wrote my first Lambda (part 1) and connected it to data store Dynamo DB (part 2).  Today I am going to use default file system for AWS system – S3.  Again using boto3 library is very handy.

Business case

To make it easy and treat DynamoDB as historical table only, I decided to make calculations based on the state written to file.  To calculate current state of the wallet I am subtracting operation amount from the last total.  And I will keep it in simple JSON on my bucket.

Reading from S3

Before I start to do the basic operations, let me mention that there are no folders on AWS file system, we have buckets here.  These are basic containers for S3 files.  Here is how we can easily read file from bucket using Pythonic Lambda.

Writing to S3

Writing is nothing simpler than using boto3.resource again. Play with it on your AWS landing page.

Permissions

Again we are to make all of our fun safe and clean.  So before running you will need to set permissions for user running lambda to use this file system.  Following policy AmazonS3FullAccess is a silver bullet to all s3 operations.

Summary

Of course it is only tip of the iceberg, when it comes to S3 file system.  It can accept any files, provides versioning and triggers, that can be leveraged to any processing I can imagine.  But what about saving simple JSON file?  Works.

AWS Lambda, part 2. Saving data in DynamoDB

The last time I was running bare Lambda function.  It was fun, but no persistence was touched.  Now if we are going to track real expenses we have to persist data somewhere.  One of the solutions made for that is to use DynamoDB table.

Czytaj dalej AWS Lambda, part 2. Saving data in DynamoDB

My First Lambda – playing with AWS, part 1

When reading through the whole internet lately I encountered AWS hype.  2015 I remember there was my first contact with this Amazon services but I skipped that somehow and forgot about my account.  Things changed when I learned that this portal allows you not only to host machines and real linux systems but Lambdas.  The code is here, check my Github.

Czytaj dalej My First Lambda – playing with AWS, part 1

Elasticsearch rozumie po polsku

Elasticsearch ma oficjalny polski plugin.  Stempel.  Udało mi się go testować i sprawdzać.  I choć polski analizator pozostawia trochę do życzenia to robi niezłą robotę.  Przypadek użycia?  Indeksujemy i wyszukujemy RSS feed z ofertami pracy ze znanego polskiego portalu.

Czytaj dalej Elasticsearch rozumie po polsku

Czytałeś bloga – napisz recenzję w 160 znakach

A więc od początku.  Wszystko zaczęło się od strony jvm-bloggers.com.  Dostrzegłem że tak na prawdę pisanie blogów to coś co możesz osiągnąć.  Ale czytanie też inspiruje!  Problem tylko taki że na internecie dzieje się bardzo dużo.  Ciężko to ogarnąć.  Dlatego wymyśliłem sobie mały zbiór opinii i recenzji, które mają pomóc podjąć mi i Tobie decyzję czy zapoznać się z danym materiałem.  Zainteresujmy się tym co na prawdę jest interesujące.  Zobacz tutaj na 160znakow.pl

Czytaj dalej Czytałeś bloga – napisz recenzję w 160 znakach

Flask can provide REST for RESTful Elasticsearch

API for ES – why

So what is the goal? I want to have small app, that will allow greedy HR workers to explore Projects I took part in. I believe, one’s experience is the most important feature of future (IT) worker; school was long time ago, courses are OK but who knows who helped to accomplish them?
Czytaj dalej Flask can provide REST for RESTful Elasticsearch

Monitoring = (Elasticsearch + Logstash + Kibana ) + Kafka * Flink

I used to be regular business developer. Delphi, then Java. Recently working somewhere, where project was not veery interesting I came to the conclusion: I would like to bring a real business value here.

Then Elasticsearch came to my mind…

Czytaj dalej Monitoring = (Elasticsearch + Logstash + Kibana ) + Kafka * Flink

Django: Web MVC and DB DAO in minutes

On Django page you can read:

Django makes it easier to build better Web apps more quickly and with less code.

I decided to validate that statement.  And yeah, after working with Django for few days it turned out to be true. Following you may see how fundamental MVC tasks were implemented by me.
Czytaj dalej Django: Web MVC and DB DAO in minutes

Data Visualization is Data Communication

Another chapter from Udacity course is data visualization.   What I have learned was very important.  To acquire, munge and analyze data.  But it would be nothing when such a information wouldn’t be communicated to others. The easiest way it is to be done with data visualization.  I learned about Napoleon’s march to Russia and why you have to think twice before you will post hue-colored diagram.

Czytaj dalej Data Visualization is Data Communication