# Processing Local YAML Files With Sidekiq > See how you can use Sidekiq jobs to read and process a local YAML template for populating the database. - Canonical: https://dennisokeeffe.com/blog/2022-04-05-processing-local-yaml-files-with-sidekiq - Published: 2022-04-05 - Updated: 2022-04-05 - Tags: ruby, ruby-on-rails This post will demonstrate how to have a Sidekiq job read a local YAML file and then take that information and add the information to a database. Source code can be found [here](https://github.com/okeeffed/demo-rails-yaml-sidekiq-job) ## Prerequisites 1. Read my previous blog post ["Getting Started With Sidekiq In Rails 7"](https://blog.dennisokeeffe.com/blog/2022-03-04-getting-started-with-sidekiq-in-rails-7). ## Getting started It is a prerequisite that you get the repo up and running by following the steps outlined in the post ["Getting Started With Sidekiq In Rails 7"](https://blog.dennisokeeffe.com/blog/2022-03-04-getting-started-with-sidekiq-in-rails-7). After you have a running Rails setup with Sidekiq, we need to add a last few things. ```s # Create a folder and file to house our YAML templates $ mkdir -p templates/yaml $ touch templates/yaml/basic.yml # Create a Person model $ rails g model Person name:string age:integer # Create a Jobs controller $ bin/rails g controller jobs create # Create a Template job $ bin/rails g sidekiq:job template # Run migrations $ bin/rails db:migrate ``` At this stage, we are ready to start updating our controllers and jobs. ## Updating our jobs controller In the file `app/controllers/jobs_controller.rb`: ```rb class JobsController < ApplicationController def create if params[:template_group].present? && params[:template_name].present? TemplateJob.perform_async(params[:template_group], params[:template_name]) render json: { message: 'Accepted' }, status: :accepted else render json: { error: 'Missing template_group or template_name' }, status: :bad_request end end end ``` Here, we effectively check that we have the required params and if so, we run the job. > Note: A better job could be done here to ensure the folder exists at this stage, but we'll assume it does for this post. Next, we need to write up our job. ## Updating our template job In the file `app/sidekiq/template_job.rb`: ```rb require 'yaml' class TemplateJob include Sidekiq::Job def perform(template_group, template_name) # Do something p "TemplateJob started with args #{template_group} #{template_name}" template = YAML.safe_load(File.read(File.join(Rails.root, 'templates', template_group, "#{template_name}.yaml"))) p "Template name: #{template['name']}" p "Template description: #{template['description']}" template['people'].each do |person| person = Person.new(name: person['name'], age: person['age']).save p "Person #{person['name']} could not be saved" if person.nil? end # Will display current time, milliseconds included p "TemplateJob #{Time.now.strftime('%F - %H:%M:%S.%L')}" end end ``` Here, we are using the `yaml` module as a helper for us to safely load the YAML file. The `template_group` and `template_name` variables are passed into the job and used to denote the name of the YAML file. Once loaded, we assume there is a `people` property which is iterated over and each person is added to the database. Before we test our job, we need to make some updates to our configuration. ## Adding in the YAML content In `templates/yaml/basic.yaml`, populate the YAML with the following: ```yaml name: Basic description: Basic YAML template to insert people people: - name: John Doe age: 30 - name: Jane Doe age: 25 - name: Joe Smith age: 32 ``` We will be using the `people` key to iterate over the people and add them to the database. ## Setting our routes In `config/routes.rb`: ```rb Rails.application.routes.draw do resources :jobs, only: [:create] end ``` ## Updating our application config In `config/application.rb`: ```rb require_relative "boot" require "rails/all" # Require the gems listed in Gemfile, including any gems # you've limited to :test, :development, or :production. Bundler.require(*Rails.groups) module DemoRailsYaml class Application < Rails::Application # Initialize configuration defaults for originally generated Rails version. config.load_defaults 6.1 config.action_controller.default_protect_from_forgery = false if ENV['RAILS_ENV'] == 'development' end end ``` We turn `default_protect_from_forgery` for this current example where I am using the HTTPie CLI to test out the job. ## Testing our job If you followed the configuration from the prerequisite correctly, then you can run `bin/dev` to start both the Rails server and the Sidekiq server. From here, I can use HTTPie to test out our job: ```s $ http POST localhost:5000/jobs template_group=yaml template_name=basic HTTP/1.1 202 Accepted Cache-Control: no-store, must-revalidate, private, max-age=0 Content-Type: application/json; charset=utf-8 Referrer-Policy: strict-origin-when-cross-origin Set-Cookie: __profilin=p%3Dt; path=/; HttpOnly; SameSite=Lax Transfer-Encoding: chunked X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: SAMEORIGIN X-MiniProfiler-Ids: w5qbp38e6xytn2a1g5kt,d6mr3lsmk8e4i8i4o6ad,ve2sy6fpwhycw1bhi8m,pk199otp1pemditsz172,zij0sg8ruw8c350s4zrd,2fu1c028w4h8j1fg8eqn,sql5jajpv0iu7zoe9z2n,lqtxoxkso0p3s9udy24i,p74qirfir3xmhhiyo31,l2h57dj2hn4g,tokv1ogolv1vcj0fxnmd,szathvf0vaxo6glg9ruw,61hri8h3fo74jhwsolez,8mt3s6max4sdgylleub4,g43fqs7faec3hts96g7e,iwfy35nvml9fbsaqzqcp,qk6mrujh9gjdg291hfjg X-MiniProfiler-Original-Cache-Control: no-cache X-Permitted-Cross-Domain-Policies: none X-Request-Id: f411f617-48eb-45e4-87d8-b48976cd121d X-Runtime: 0.045609 X-XSS-Protection: 1; mode=block { "message": "Accepted" } ``` Once the job is accepted, it will be queued and run by Sidekiq. After the job completes, we can check the results in the Rails console: ```s $ rails c irb(main):001:0> Person.all (1.1ms) SELECT sqlite_version(*) Person Load (0.2ms) SELECT "people".* FROM "people" => [#, #, #] ``` Success! ## Summary Today's post demonstrated how to read a local YAML file from a Sidekiq job and use the data from the file to populate a particular model in the database with some entries. ## Resources and further reading - ["Getting Started With Sidekiq In Rails 7"](https://blog.dennisokeeffe.com/blog/2022-03-04-getting-started-with-sidekiq-in-rails-7) - [Source code](https://github.com/okeeffed/demo-rails-yaml-sidekiq-job) _Photo credit: [`pawel_czerwinski`](https://unsplash.com/@pawel_czerwinski)_