There are tons of methods for organising binaries into a particular project in different programming languages. But, I am writing this method particularly to illustrate the pure usage of elixir and joy of learning it! This tutorial is for intermediate users with enough experience in using elixir. I hope you enjoy it!

How can you leverage it?

It could be anyone of the following:

• a mix task
• a genserver
• a function
• a escript etc.,

In my particular use case, I wanted a mix task, so I will be illustrating the use case with Mix Tasks. Feel free to try and explore different options.

Mix Task:

We all would have started using mix tasks, the moment we installed elixir in our system. The basic mix task that helped me my elixir journey is mix help. The first thing that I wanted to learn is how to create a simple mix task. I am not going to ellaborate on how to write a mix task, but rather focus on the main functionality.

Requirements:

• Erlang ~> 20
• Elixir ~> 1.6
• A simple elixir project (create it using mix new)
• Create a basic mix task (Reference)

Getting Started

# Your example mix task should be like the one below
defmodule Mix.Tasks.Hello do
  @shortdoc ~S("Example mix task for showing hello world")
  
  use Mix.Task
  def run(_) do
  #################
  Code hidden......
  #################
  end
end

Add {:httpoison, "~> 1.3"} to your new sample project’s deps in mix.exs. You can get to know the latest version of HTTpoison using mix hex.info httpoison.

Writing a Basic Download Service using HTTPoison

First of all you need to know how much would it take for the download to complete. As we are not creating something competing to wget, curl or even IDM / XDM, I will keep it very simple, I am going to download something small with a timeout of 5 minutes, so that my task will wait for 5 minutes for the download to complete.

• A dummy download link
• Timeout: 5 minutes (5 * 60 * 1000)

With the above things ready, we are nearly ready to get into coding. Make sure to either keep the download url in config or use the OptionsParser in Elixir.

def download(url, path) do
  {:ok, file} = File.open("path/filename", [:write, :exclusive])
    ...to continue
end

One main thing to remember is that, since it is a mix task, you are supposed to start HTTpoison before starting the download. If you are writing a mix application, just starting HTTPoison in the mix.exs is enough.

def download(url, path) do
  HTTPoison.start() #Mandatory if you are using it in a mix task
  {:ok, file} = File.open("path/filename", [:write, :exclusive])
end

Using HTTPoison get

HTTPoison.get(url, ["Accept": "application/octet-stream"], [follow_redirect: true, stream_to: self(), recv_timeout: 5 * 60 * 1000])

When you just need to hit an endpoint, the syntax is as simple as HTTPoison.get(url, []) but since, we want to download, I am using the necessary headers and follow_redirect is optional because, some sites redirect you to cloud storage services, where you will get the file and you can also control, you redirection limit. The stream_to is the interesting part, which is the thing that let’s us download without blocking the IO. So, the above command gives a tuple in the form {:ok, %HTTPoison.AsyncResponse{id: ref}}. The tuple contains a reference to the downloading process.

Now the code will look like the following:

def download(url, path) do
  HTTPoison.start()
  {:ok, file} = File.open("path/filename", [:write, :exclusive])
  with {:ok, %HTTPoison.AsyncResponse{id: ref}} <- HTTPoison.get(url, ["Accept": "application/octet-stream"],
                [follow_redirect: true, stream_to: self(), recv_timeout: 5 * 60 * 1000]) do
    write_data(ref, file, path)
  else
    error ->
      File.close(file)
      File.rm_rf!("path/filename")
  end
end

Further steps

The HTTPoison provides certain structs to understand the response from the server it hits and provides easier interface to interpret each type of struct and consume them.

alias HTTPoison.{AsyncHeaders, AsyncRedirect, AsyncResponse, AsyncStatus, AsyncChunk, AysncEnd}

The above aliases and our function in the previous snippet write_data(ref, file, path) are closely related.

defp write_data(ref, file, path) do
  receive do
    %AsyncStatus{code: 200} ->
      write(ref, file, path)
    %AsyncStatus{code: error_code} ->
      File.close(file)
    %AsyncRedirect{headers: headers, to: to, id: ^ref} ->
      download(to, path)
    %AsyncChunk{chunk: chunk, id: ^ref} ->
      IO.binwrite(file, chunk)
      write(ref, file, path)
    %AsyncEnd{id: ^ref} ->
      File.close(file)
  end
end

The write_data is just a receive do loop with pattern matching.

• first pattern checks for connection success code 200 and then it again sends to the same function
• the next, it checks for codes other than success and closes the file if it is executed
• the redirect pattern, checks for any redirects and get the redirect url and starts the download with the new URL
• the chunk is the real hero of the day, which accumulates our downloaded data in chunks and writes to the specified file
• the end indicates the completion of streaming from the process.

Finally our download task is ready

defmodule Mix.Tasks.Hello do
  @shortdoc ~S("Example mix task for showing hello world")
  use Mix.Task
  alias HTTPoison.{AsyncHeaders, AsyncRedirect, AsyncResponse, AsyncStatus, AsyncChunk, AysncEnd}
  def run(_) do
  download("#{URL}", "#{SAVE_TO_PATH}")
  end
  def download(url, path) do
    HTTPoison.start()
    {:ok, file} = File.open("path/filename", [:write, :exclusive])
    with {:ok, %HTTPoison.AsyncResponse{id: ref}} <- HTTPoison.get(url, ["Accept": "application/octet-stream"],
                  [follow_redirect: true, stream_to: self(), recv_timeout: 5 * 60 * 1000]) do
      write_data(ref, file, path)
    else
      error ->
        File.close(file)
        File.rm_rf!("path/filename")
    end
  end
  defp write_data(ref, file, path) do
    receive do
      %AsyncStatus{code: 200} ->
        write(ref, file, path)
      %AsyncStatus{code: error_code} ->
        File.close(file)
      %AsyncRedirect{headers: headers, to: to, id: ^ref} ->
        download(to, path)
      %AsyncChunk{chunk: chunk, id: ^ref} ->
        IO.binwrite(file, chunk)
        write(ref, file, path)
      %AsyncEnd{id: ^ref} ->
        File.close(file)
    end
  end
end

This is our basic downloader and is ready for use. Please, keep in mind the above code is not production ready, and don’t put system critical uses using the above sample code, but it is usable after some mission critical tweaks. All the best!