Python Script to Read CSV in S3 Bucket and Upload Data to DynamoDB Table

Python Script/Code to Read CSV and Upload Data to DynamoDB Table

import boto3
import csv
import json
import pandas as pd
import decimal
access_key='ABCDEFGHIKLMNOPQR'
secret_key='ZYXVUABCDEF+-GHIJKL'
region='ap-south-1'
session = boto3.Session(aws_access_key_id=access_key,aws_secret_access_key=secret_key,region_name=region)
s3 = session.resource('s3')
file_url = s3.Bucket('source-bucket104').download_file('prices.csv', 'local_prices.csv')
dataframe=pd.read_csv('local_prices.csv')
db1=boto3.resource('dynamodb',aws_access_key_id=access_key,aws_secret_access_key=secret_key,region_name=region)
table=db1.Table('prices')
cnt=len(dataframe.index)
for i in range(cnt):
  row=dataframe.head(i+1).tail(1).to_json(orient='records', lines=True)
  row_dict=json.loads(row,parse_float = decimal.Decimal)
  table.put_item(Item=row_dict)

Comments

Configure Oracle ASM Disks on AIX

Configure Oracle ASM Disks on AIX You can use below steps to configure the new disks for ASM after the raw disks are added to your AIX server by your System/Infrastructure team experts: # /usr/sbin/lsdev -Cc disk The output from this command is similar to the following: hdisk9 Available 02-T1-01 PURE MPIO Drive (Fibre) hdisk10 Available 02-T1-01 PURE MPIO Drive (Fibre) If the new disks are not listed as available, then use the below command to configure the new disks. # /usr/sbin/cfgmgr Enter the following command to identify the device names for the physical disks that you want to use: # /usr/sbin/lspv | grep -i none This command displays information similar to the following for each disk that is not configured in a volume group: hdisk9 0000014652369872 None In the above example hdisk9 is the device name and 0000014652369872 is the physical volume ID (PVID). The disks that you want to use may have a PVID, but they must not belong to a volu...

Adding New Disks to Existing ASM Disk Group

Add Disks to Existing ASM Disk Group In this blog I will show how to add new disks to an existing ASM Disk group. This also contains the steps to perform the migration from existing to the new storage system. In order to add the disk to the ASM disk group, you will first need to configure these disk using the operating system commands. I have provided the steps to configure the disks on AIX system in my blog " Configure Oracle ASM Disks on AIX" Adding New Disks to DATA Disk Group (Storage Migration for DATA Disk Group) Login to your ASM instance $ sqlplus / as sysasm If the name of the new disk is in different format from the existing disk, the modify the asm_diskstring parameter to identify the new disks. In my below example /dev/ora_data* is the format of the existing disks and /dev/new_disk* is the naming format of the newly configured disks. You should not modify this parameter unless the naming format changes. SQL> alter system set asm_diskstring = '/dev/ora_data*...

Load records from csv file in S3 file to RDS MySQL database using AWS Data Pipeline

In this post we will see how to create a data pipeline in AWS which picks data from S3 csv file and inserts records in RDS MySQL table. I am using below csv file which contains a list of passengers. CSV Data stored in the file Passenger.csv Upload Passenger.csv file to S3 bucket using AWS ClI In below screenshot I am connecting the RDS MySQL instance I have created in AWS and the definition of the table that I have created in the database testdb. Once we have uploaded the csv file we will create the data pipeline. There are 2 ways to create the pipeline. Using "Import Definition" option under AWS console. We can use import definition option while creating the new pipeline. This would need a json file which contains the definition of the pipeline in the json format. You can use my Github link below to download the JSON definition: JSON Definition to create the Data Pipeline Using "Edit Architect" ...

Akshay Narang

Search This Blog