temp files will be stored and deleted after processedįilename = file.split("/")#get file name Home = os.path.expanduser("~")#get path of the home directory of the user. Input_lete_path(file)#delete file from input_folder With input_folder.get_download_stream(file) as f:#read file from managed folder Output_folder = dataiku.Folder("managed_folder_out_id")įor file in input_folder.list_paths_in_partition(): Input_folder = dataiku.Folder("managed_folder_in_id") Below is the code that I tested in my DSS instance using Python notebook and Scenarios: import os The other approach you can try to utilize the FTP connection you have and Dataiku API.Īs a prerequisite, you should have two managed folders setup using your FTP connection and the files you want to process and move located in one managed folder so you can process, zip and move them to another managed folder. I tested it in my DSS python notebook on my test instace, and it works fine for me. Hi if your DSS FTP connection working, it means the instance has access to FTP server so the above code should work. If not, and I misunderstood your question, please provide your use case to understand the requirements better. #dataiku api, pandas or any other python package can be used here if you need to performe any data manipulation and/or create a dataset from the fileįtp.storbinary('STOR '+filename, open(local_filename, 'rb')) Local_filename = os.path.join("/tmp", filename)įtp.retrbinary("RETR " + filename, lf.write, 8*1024) You could try using the ftplib module ( ) or some other similar python package from doing a quick search online.įor example, with ftplib the file can be download to the local /tmp folder, removed from the original FTP folder, zipped and then uploaded to another FTP folder.īelow is a test I run in Python notebook on my test instance: from ftplib import FTP If I understood correctly and the requirement is to access some files in one FTP folder, archive them and move to another FTP folder, then it isn't DSS specific but more a Python exercise. Developer Guide Tutorials and articles for developers and coder users →.Knowledge Base Articles and tutorials on Dataiku features →.Reference Documentation Comprehensive specifications of Dataiku →.Discover the winners and finalists of the 2023 edition, and read their story to learn about their pioneering achievements in data science and AI! View Winners and Finalists.Explore the Community Discover, share, and contribute →.Join the Academy Benefit from guided learning opportunities →.Partner Back Partner Library Partner Discussions Partner Events Partner Demand Center Partner Administration Partner Marketing
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |