Delete some specific objects from a JSON file using python and then push the file to Elasticsearch Index -
i having large json file contains many object. trying remove number objects fetched file since contains public keys , stuff.. ref have named download.json:
"osfamily": "debian", "sshrsakey":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx2xxx", "lsbmajdistrelease": "32", "interfaces": "eth0,lo", "physicalprocessorcount": 1, "ec2_kernel_id": "pki-724545-724545",
my python script downloads above json file sqs queue , pushes es server follows :
import os import json import uuid import time import boto.sqs import boto boto.sqs.connection import sqsconnection boto.sqs.message import message boto.sqs.message import rawmessage sqs = boto.sqs.connect_to_region("ap-southeast-1") q = sqs.get_queue("nishantqueue") #text_file = open('download.json', 'w') m = q.read(visibility_timeout=15) if m == none: print "no message!" else: open('download.json', 'w') json_data: print m.get_body() json_data.write(m.get_body()) # clean_data = json.load(json_data) ## # element in clean_data: ## # del element['sshdsakey','sshrsakey'] # json_data.write(clean_data) ## json_data.close() q.delete_message(m) print "push es" os.system('./push_to_es.sh') print "cleaning temporary json file" os.remove('download.json') print "++++++ successful run +++++++"
now can see in commented part, trying delete objects don't want , write download.json file , push es . commented part not working way thinking should , throws : ioerror: file not open reading
any appreciated
open file in read write mode.
change:-
with open('download.json', 'w') json_data:
to
with open('download.json', 'r+') json_data:
-------------edit-----------------
why not following:-
clean_data = json.loads(m.get_body()) element in clean_data: del element['sshdsakey','sshrsakey'] #write clean_data download.json json_data.write(clean_data) q.delete_message(m) print "push es" os.system('./push_to_es.sh') print "cleaning temporary json file" os.remove('download.json') print "++++++ successful run +++++++"
Comments
Post a Comment