DevOps and Linux practice reflection

During this month I have been practicing some common DevOps activities like setting up a repository, opening a server and updating it automatically with cron. Some of them were easy tasks because I’ve already done similar activities, but there were other task that I had no idea how to do and it took me a while to figure it out.

If you want to see the whole process I’ll link the part 1, part 2 and part 3 here. On this post I’m going to talk about my experience doing this activities and then show the results.

The first part was easy, most of the step were things I already had like the Linux distribution(Ubuntu 16) and all the development environment. I had to struggle a little bit with cron because I din’t know how to make sure it was running, but after reading a couple of tutorials I found that I had to send the output to a file with the «>>» symbol.  This operator appends anything I pass to the file without eliminating what was there before. This is an example of how my cron file looks:

* * * * * [path]/bashscript >> [path]/log.out

For the second part I also had experience using SSH keys for GitHub, I use that all the time now that I have 2FA because for the terminal it won’t let me use my normal password. To code the server I used NodeJS and ExpressJS because those were tools I was familiar with for this kind of things, later I had trouble with that but for this part it did what it had to do, the script moves to the git folder and pulled from master. Just to keep track of what was happening I kept the log file. During a period of time the script created a file and after doing all the stuff it had to do deleted it, that was surrounded with an if conditional that checked if the file was created. I deleted that on part 3 to simplify my tests because while I was working on it I had no idea what was giving me errors.

Finally, the part 3 was the most difficult for me. I quickly created a Maven project and in 5 minutes I could run the test without problems. The tricky part for me was that I could not find the way to read a file with the tests results and pass that to the web page. I tried a lot of ways and after a lot of failures and attempts to debug my code I learned that ExpressJS was not made to host HTML files. My HTML contained the JS that reads the tests results, that JS script was run in an environment without the proper modules so it didn’t work. My solution was to format the tests results files to have the proper HTML tags and then put that into a static HTML. This is how it looks:


It’s not pretty, but is useful…I guess. The code of the practices is available here

Overall, I enjoyed learning and using things that I don’t normally use, like cron or bash scripts. I think these tools are really helpful and I would like to learn more because I could find more interesting uses to it. After doing a little bit of research I found tools that could’ve helped me to do this easier. Jenkins is a tool that automates the whole development process, it’s open source and free. Kubernetes is another tool for automation but this one works for containers, from what I’ve heard is harder to learn and it’s not free. Both of them use Docker, a tool to containerize apps that I’ve hear a lot but haven’t had the chance to use it and learn why is useful. I think that all these tools are really powerful but sometimes complex, I need to learn how to use them well before implementing anything with them.

Now I know that doing all these tasks can be a lot of work and requires a lot of knowledge and experience to do some high quality automation.  Being a DevOp also requires a lot of creativity, each environment is different in hardware, software and people, the DevOp has to have a different set of abilities to solve each problem, whether it is about automation scripting or people management.

Anuncio publicitario

DevOps and Linux practice pt. 3

Continuing from the DevOps and Linux practice pt.2, this time I’m going to use Cron to run the tests automatically and update the web site based on the results.

First I had to start a new Java project in the repository, after reading a little bit I saw that running tests from console was easier if I used Maven. So I first installed Maven using sudo apt install maven and then started a Maven project with VSCode.  To set up maven correctly and run my first tests with it I followed this tutorial.

The test results will be exported to a file so that the web page could read it and update the info in it. For this part I had a lot of issues trying to read a file and pass the text to the HTML, I could not run a reader script from the HTML because ExpressJS is not made for client applications. A simple solution for this is to write the output as if it were a HTML file and then just host that new HTML. Before rewriting the new file I formatted the output with regular expressions firsts in bash.

All this work took me a lot of time to do, especially the part where I had to show the testing results in the page somehow. I tried a lot of methods I found on Internet and I ended doing a mixture of them to do something I was happy with. At the end of this week I’m going to write more about this and show the final results.


Reflection about Blogging and Me

The Bloggin and Me blog post by Ana is an insightful and fun read. She writes about a lot of topics that not many people want to talk about like personal fears and life problems, but she does it in a very relaxed style with some jokes. I like to read about others people’s life because sometimes I find that I’m not very different to others and I can relate a lot of their thoughts and experiences to mine.

I can imagine how she felt when she stopped having fun for working and earning money. I’ve always been very curious about technology, science and engineering since I was a kid, I liked to be creative, build stuff, experiment with new knowledge and have fun doing anything non-school related. I liked to challenge myself and discover or learn new things along the way. However, I stopped feeling like that since my mid years in college. Now, I rarely feel like I want to do something new and interesting, I don’t feel that fresh inspiration that pushes me to try something new or stupid or just fun. I know is not that I don’t have time, I have a lot of free time, I just don’t feel like that anymore. I spent most of my free time playing video games or watching series, tired, maybe thinking about the next assignment, the next exam, the grades or that I should sleep more. Now I see those fun activities as a waste of time and unproductive, but then I proceed to do something even less productive or worst, less fulfilling. Perhaps I’m burned out by school, I’m forced to think that I should spend my free time doing productive stuff to be more competent in my area, but that thinking makes me get tired quicker, more stressed and with the constant and unfulfilled desire to rest. I feel like when I was younger when I do challenging homework that pushes me to be creative, but each time is harder to find the right spot of time to start the work and put a decent amount of thinking to it. I mostly do homework to finish it, not to learn from it, hardly I sit and take my time, I don’t have fun anymore, and that is sad.

It is very inspiring how she quickly started to make conscience about her own thoughts and feelings to start working over them. She realized that a lot of people around her were toxic, and that most of her fears were a common occurrence among the developer community. In her words «Your fears are part of who you are and use them to your benefit». This line made me think about a conference where Jürgen Klaric talked about fear, and how fear forges our experience as human being. Fear is good because it saves us from apparent danger, it’s an instinct that is hard to avoid and very natural to follow. But fear is what stop us from doing more and better things with our lifes. He mentioned that eliminating our fears is impossible, fear is always going to be there to warn us, we cannot avoid it nor eliminate it, but we can learn to take advantage of it by understanding it.

At the end she mentions the personal benefits of having a space for you in your blog, more that doing it for someone else, is doing it for you. I think that having a space just for me and my silly thoughts, record my new learning or posting any content would very nice to have, but I just don’t find the motivation to do it. I hope that somewhere in the near future I can get healed from this weird mental illness and finally enjoy life as I used to, without worrying about grades, without worrying about how much I earn, without worrying about how «productive» I am and without worrying about what everyone else thinks of me. I think that maybe giving me the space to write a blog about anything could work as meditation and a relaxation activity. Maybe I could try it in the future when I’m done with the semester. I’m of course doing what I can to enjoy every second of my life and my beautiful student life, but is still hard sometimes to get out of the comfort zone and have enough energy to do something for myself.

DevOps and Linux practice pt. 2

After setting up our development environment and practicing a little bit with cron in part 1, it’s time to do something useful with it. But first, lets make sure that my repository and account are safe.

1.- Setting up 2FA

GitHub has the option to connect with 2 factor authentication to your account, I have activated this option and now after signing up in an unknown device they ask for a number that they send to my phone via SMS. Now, whenever I try to push from the terminal when I’m not logged in it won’t accept my password account, this happens because with 2FA I have to generate a token and use that token as my password. I use two tokens, one for Linux and the other for Windows, they both have the minimal permissions to let me push into the repository but not delete or modify anything important from that repository or others as well.

2.- Setting up SHH keys

To avoid copying and pasting that token each time I want to use my account in the terminal there is a alternative, the SSH keys. To generate one I followed this complete tutorial, now I can connect with the key from any place I want.

3.- Cron server update automation

Now for the cron practice, I already changed the bash script to do a pull instead of just printing hello world. Also, I considered to change the cron file to run the script every 12 hours. I’m still not sure how often I should update, or how often companies do it in a real life development environment, but 12 hours seemed reasonable for me because it updates the server when people(me) are supposedly not updating the repository. However, the problem with this is that if someone pushes a broken piece of code, it will not be fixed until the next 12 hours or that someone corrects the bug and pulls manually from the server. For this practice I need to watch consistent changes, so I’m going to change the script to do a pull each minute. Whenever the server pulls any change, the website updates with a refresh if it is a simple change like a text line in the html, but I need to run it again if something like new dependencies and a JS script were added. I’m not sure if this happens because I’m using NodeJS to run the server or if this is a normal behavior, but Node already has some solutions related to this, check this site if you want to see how to use the node demon or you want to avoid refreshing the web page each time the server updates. I’m not implementing any of this for now, but I will consider it for the next practices. To ensure that the server is not running two copies of the same script there are several techniques, a simple solution is to create a file and after making the pull deleting the file. Before creating the file we should check if it already exists, if it does, then wait until the script that created it deletes it; if it doesn’t exists; then there is no concurrency and everything should execute normally.  I added this solution in the bash script, to do that I had to learn first how to do it. This bash cheat sheet helped me a lot, I think is a very useful resource when you are coding scripts.

Python unit tests

Today’s activity is really interesting, we have to do some practice with python unit testing. According to the official unittest documentation, it’s very similar to JUnit, so we should have no problem understanding it.

Now, here is a picture of me using hypothesis to do annotations on an old web page that we are able to see thanks to Wayback Machine.


While playing a little bit with Wayback Machine I found an old version of Steam’s page, the popular PC gaming platform, back in 2002. Its interesting to see that web pages were not that colorful or filled with images, this design was simple but worked really well.


For the next part, we have to see this Unit Testing and Test Driven Development in Python course where we will learn to set up virtual environments and unit tests using Pytest. For the exercises we are going to use PyCharm, a Python IDE that will make easier for us to run tests.


The tutorial was fun and explained very well the basic functionalities that Pytest has. The activities were helpful summarize all the material and I felt that they were easy to do and understand.  Overall, Pytest is a very simple but powerful testing framework for Python, it’s really easy to use and has a lot of functionalities that solve testing problems like mocking objects and exceptions catch. It’s easy to setup as well, thanks to PyCharm I had no trouble at all when I was implementing my first tests. All the time I felt like I was using JUnit but with different syntax. At the end, Richard Wells also explained some TDD good practices, and one that called my attention was to run tests in random order, I hadn’t thought that it’s important to make sure that tests should run in any order which means that they don’t have any dependencies between each other. For other specific testing processes like coverage tools, advanced mocking or linting there are other libraries or modules that work very well with Pytest.

Chapter 6 activity

What two properties must be satisfied for an input domain to be properly partitioned? 

– The partition must cover the entire domain (completeness)
– The blocks must not overlap (disjoint)

What is an Input Domain Model (IDM)? 

An input domain model (IDM) represents the input space of the system under the test in an abstract way. It helps the test engineer to define and create better tests by establishing the structure of the input domain and partitions of this domain.

What gives more tests, each choice coverage or pair-wise coverage?

Pair wise, as it does a permutation for a value from each block with one of each block of the other characteristics.

DevOps and Linux practice

This week we are going to learn about Development Operations. Its a topic that I hear very often in the industry and I didn’t know really well what this was about. If you want to learn more about what DevOps is, I made a blog post summarizing a reading about that here.

For this blog I’m going to be completing some task that show the job of a Dev Ops engineer in a simple way. For each achievement I’m going to talk about my experience:

1.- Install a Linux distribution

I already have a partition of Ubuntu Desktop 16 in my laptop, I use it for some classes or when I need to go full developer mode(no video games). I installed it a while ago, I chose Ubuntu because it is a popular choice for newcomers and students. It’s easy to customize and looks pretty, there might be better choices but at the moment I don’t have the necessity to change my Linux distribution.

2.- Install support for your development environment.

This task is really important, here is where we chose our tools, programming languages, text editors, libraries and everything else we need to have our development environment ready to go. Most of the choices I made here were just determined by preference. It’s important too that you customize your environment as you want, choose the text editor you are most used to, the tools you prefer and the themes you like; remember that you are most likely be working here a lot of time, so it does not hurt to make it easier for you.

Languages: I already have Python 2 and 3, Java, C and C++. That includes any compiler or installation needed.

Code editor: For big projects I installed Visual Studio Code, it’s a really robust editor with a lot of functionalities and a lot of extensions created by the community. For quick edits I like to use Sublime, it may have less functionalities than VSCode but is faster and also looks pretty.

Tools: I installed Git for version control, Conda to handle several environments and NodeJS to make Web Applications easily.

3.- Setup your first test use of Cron on your server.

Crontab is a very useful tool to run scripts at specific times. It’s really simple, this tutorial is enough to understand how to create a crontab file. For this practice I made a bash script that prints «Hello world» and runs every 5 minutes.

"gears" flickr photo by el_rogos shared under a Creative Commons (BY-NC-ND) license

Development Operations

Development Operations, or DevOps, is a work philosophy that is based on the thinking that «software isn’t done until it’s successfully delivered to a user and meets their expectations around availability, performance, and pace of change»(Mueller, 2010). A lot of people see it as a job title where the person with the job has the responsibility of building the infrastructure for development and make sure the software product is delivered in the right way. As the What is DevOps? blog post explains, it’s just more than that. DevOps role is perhaps misunderstood as «the guy who tell us to use docker and chef»; as I said in the beginning, it’s an entire work philosophy that picks the main ideas from Agile development and transforms them into a set of practices, tools and principles to deliver a product continuously.

In past years DevOps was not even a thing because no industry needed that, but that changed with the software demand explosion. Now software is not a product that is delivered once and that’s it, we consume software as a continuous service that needs maintenance, adjustments and changes on the run.  Developers now need to add new features to a software that has already launched and ten thousand people are using at the same time. The goal of DevOps is to solve this problem with automation to make it easier and reduce error. This overall means that DevOps is about Infrastructure Automation, Continuous Delivery and Site Reliability Engineering as Mueller suggests.

Doing a little bit of research I found a REALLY good video about DevOps that focuses a little more on explaining what is the DevOps mindset and why it’s important to understand it. Andrew Clay Shafer is a DevOps evangelist that visited Guadalajara a week ago to talk about his passion, I didn’t have the chance to go to the conference but this video might be the equivalent:

«DevOps is optimizing human performance and experience operating software…with software… and humans» (Shafer, 2018)

One interesting idea that I picked from the video is that organizations and humans need to learn to improve and become more productive, and to learn we all must be able to make mistakes and change our behavior. DevOps is a culture of continuous improvement, where different persons with different backgrounds and knowledge gather together to reach the same goal, that is to get better.

Chapter 4 and 5 activity

After reading chapter 4 and 5 from Introduction to Software Testing, we were asked to answer these questions about TDD and test coverage criteria:

  1. What is “correctness” in agile processes?
    Correctness for testing and software engineers has to do with the quantity of green tests, this means that our code is correct if it passes all the tests.
  2. Do TDD tests do a good job testing the software?
    No, TDD tests are mainly to define the requirements and specifications of our code, but not really to evaluate the right behavior or to cover edge cases.
  3. Can we automate our tests without TDD?
    Yes, automation can be done in test without following the TDD principles. Imagine writing first the code, then just executing the tests to validate it.
  4. Can we use TDD without automating our tests?
    Yes, TDD consists on putting the tests first as a guide for the code to be written, automation can help to write tests and validate faster, but is not mandatory.
  5. What four structures do we use for test criteria?
    1.- Test Requirement: A test requirement is a specific element of a software artifact that a test case must satisfy or cover.
    2.- Coverage Criterion: A coverage criterion is a rule or a collection of rules that impose test requirements on a test set.
    3.- Minimal Test Set: Test set T such that if a test was removed from T, it no longer satisfies all test requirements.
    4.- Minimum Test Set: Smallest possible test set T that satisfies all test requirements.
  6. What usually prevents our tests from achieving 100% coverage?
    What usually prevents our tests from achieving 100% coverage?
    Test requirements that cannot be satisfied are called infeasible. Formally, no test case values exist that meet the test requirements. The detection of infeasible test requirements is formally undecidable for most coverage criteria, and even though researchers try to find partial solutions, they have had only limited success. Thus, 100% coverage is impossible in practice.

  7. Some organizations in industry who adopt TDD report that it succeeds very well, and others report that it fails. Based on your knowledge of TDD and any experience you have, why do you think it succeeds sometimes but not all?
    Companies does not use TDD well, that’s why they don’t succeed using it. It is hard for some engineers to follow TDD principles, some of us are not used to write the tests first before the code.
  8. A few software organizations use test criteria and report great success. However, most organizations do not currently use test criteria. Based on your knowledge and experience, why do you think test criteria are not used more?
    We think that test criteria is very ambiguous, as we don’t have an exact default way of measuring test criteria, everyone does it differently. Maybe having a high test coverage is expensive and most of the times is not worth it, that’s why most companies decide to not focus on having  heavy testing development.

The Secret Life of Bugs Opinion

After reading the paper «The Secret Life of Bugs: Going Past the Errors and Omissions in Software Repositories » I realized that debugging is still a big problem for software engineers and even for teams inside big companies. I imagined that everyone already have a predefined way to deal with bugs, but the paper shows that each company and team has their own methods, and not every method is very successful.

One of the many ways of dealing with bugs according to the research is to have a bug data base that keeps track of the first appearance of the bug, a description, how to solve it and the day the issue was closed. This process was many times automated and that lead to many errors like incomplete data, repeated information or erroneous information. In many cases people adopted insufficient practices due to the lack of correct information, for example, trying to find the bug owner sending emails to people they might thing have something to do with it. This happened because the database didn’t have a correct owner specified, in many cases the person listed there had nothing to do with the bug.

The paper suggest that debugging is not an process that can be automated because it depends on social and organizational factors. People need to communicate their knowledge and experiences in order to know exactly who found the bug and who can solve it. This research has 10 years now, can be considered old by many people; however, I don’t think the research and information discussed here is irrelevant to actual times.  A lot of companies and engineers can learn something from here, debugging is often a process that takes a lot of time and that is why we try to automate it, but maybe the faster way to do it is by talking to the team and looking for experts to solve it personally.