Uploading Files Locally Using Rest (Linux)
In Sisense Linux distribution you are able to upload files to the Sisense environment via the file management Interface. This article describes few ways for uploading files programmatically using bash, Powershell and python. Powershell Script for uploading a file via rest api for Sisense version greater than L2021.5 # ------- Config ------- # Sisense base DNS # For example http://23.50.45.130:30845 $baseUrl="<Sisense DNS>" # Destination location on sisense server - must be accessible via Sisense file management interface $remoteLocation="/test1/data.csv" # Local file path $filePath="data.csv" # admin api token $apiToken="<Admin API Token>" # Should the new file override existing files with the same name $shouldOverride="true" # ------- End Of Config ------- # File Management URL $uploadPathUrl="/app/explore/api/resources/data" # Create URL to send file to $url=$baseUrl+$uploadPathUrl+$remoteLocation + "?override=" + $shouldOverride #Write-Output $url # Create authorization bearer header $authorization="Bearer $apiToken" #Write-Output $authorization # In order to add files via the file explorer, one requires also x-auth token. the next part fetchs x-auth using the admin api token $xauthpath="/app/explore/api/login" $xauthurl=$baseUrl+$xauthpath #Write-Output $xauthurl $headers = @{ 'Authorization' = $authorization } # Send request to get xauth # For https remove -AllowUnencryptedAuthentication $xauth = Invoke-RestMethod -Uri $xauthurl -Method Post -Headers $headers -AllowUnencryptedAuthentication #Write-Output $xauth $headers += @{ 'X-Auth' = $xauth } # Send a request to upload a file # For https remove -AllowUnencryptedAuthentication $res = Invoke-RestMethod -Uri $url -Method Post -Headers $headers -AllowUnencryptedAuthentication -ContentType 'text/csv' -InFile $filePath # Print response Write-Output $res Bash Script for uploading a file via rest api (using cUrl) for Sisense version greater than L2021.5 #!/bin/bash # ------- Config ------- # Sisense base DNS # For example http://23.50.45.130:30845 baseUrl="<Sisense DNS>" # Destination location on sisense server - must be accessible via droppy remoteLocation="/test1/data.csv" # Local file path filePath="data.csv" # admin api token apiToken="<Admin API Token>" # Should the new file override existing files with the same name shouldOverride="true" # ------- End Of Config ------- # File Management URL uploadPathUrl="/app/explore/api/resources/data" # Create URL to send file to url=$baseUrl$uploadPathUrl$remoteLocation"?override="$shouldOverride echo $url # Create authorization bearer header authorization="Bearer $apiToken" # In order to add files via the file explorer, one requires also x-auth token. the next part fetchs x-auth using the admin api token xauthpath="/app/explore/api/login" xauthurl=$baseUrl$xauthpath echo $xauthurl # Send request to get xauth xauth=$(curl -H "Authorization: $authorization" $xauthurl) echo $xauth # Send a request to upload a file res=$(curl -H "Authorization: $authorization" -H "X-Auth: $xauth" -H "Content-Type: text/csv" -d @$filePath $url) # Print response echo $res Power-Shell 1 file upload Example - Version <=L20121.3 The script below will allow the user to upload files programmatically, using REST command. The script itself is written in Powershell. Change the first 4 parameters dns - The URL you use for your Sisense site RemoteLocation - The location of the target file within File Manager FilePath - The local location of the file from the Local Server AUTH_TOEN - Generate a token and replace it with <token> $dns= 'https://myhost.sisense.com' $RemoteLocation= 'data/Test' $FilePath = 'C:\myfolder\myfile.csv'; $AUTH_TOKEN = 'Bearer <Token>' ############################################################### $pos = $FilePath.LastIndexOf("\") $URL = $dns+'/app/explore/!/upload?vId=0&rename=0&to=/'+$RemoteLocation; $filename = $FilePath.Substring($pos+1) $fileBytes = [System.IO.File]::ReadAllBytes($FilePath); $fileEnc = [System.Text.Encoding]::GetEncoding('UTF-8').GetString($fileBytes); $boundary = [System.Guid]::NewGuid().ToString(); $LF = "`r`n"; $Headers = @{'Authorization'=$AUTH_TOKEN}; $bodyLines = ( "--$boundary", "Content-Disposition: form-data; name=`"filename`"; filename=`"$filename`"", "Content-Type: application/octet-stream$LF", $fileEnc, "--$boundary--$LF" ) -join $LF Invoke-RestMethod -Uri $URL -Method Post -ContentType "multipart/form-data; boundary=`"$boundary`"" -headers $Headers -Body $bodyLines Python Example For Folder Syncing (Upload Only) In this example you can set a local folder with sub-folders to sync with a File Management virtual library. First run will upload and update all files and create a lastRunTime.json file for reference of last run. On the next run only files that the update date is larger than the last run time of the application will be uploaded/updated Config.ini file sample: [DEFAULT] # Sisense URL host = https://test.sisense.com # Path to store files on target machine remoteLocation= data/Test # Path to folder or file which should be transferred # If folder is specifeed, all subfolders will be also proccessed path = C:\Users\Documents\Test\ # Sisense API token token = Bearer TOKEN # Script will store last modified time for each processed file. This will allow it to upload only modified files on next executions lastRunFilename = lastRunTime.json Droppy.py file sample: Droppy.py example Version <=L2021.3 import uuid import requests import io import json import os import time from datetime import datetime import configparser config = configparser.ConfigParser() config.read('config.ini') lastRunFilename = config.get('DEFAULT','lastRunFilename') host = config.get('DEFAULT','host') remoteLocation = config.get('DEFAULT','remoteLocation') path = config.get('DEFAULT','path') token = config.get('DEFAULT','token') # Cut slashes at the end of paths if they are exist if path[-1] == '\\': path = path[:-1] if remoteLocation[-1] == '/': remoteLocation = remoteLocation[:-1] # Function to compare stored and file last modified time def compareModificationTime(filePath): if (filePath in lastRunTime and os.path.getmtime(filePath)>int(lastRunTime.get(filePath))) or (filePath not in lastRunTime): return True else: return False # Function to upload file to Droppy via Sisense API def uploadFile (filesNames, folderPath): if os.path.isfile(path): globalFolderPath = os.path.dirname(path) else: globalFolderPath = path if folderPath != globalFolderPath: remotePath = remoteLocation + folderPath.replace(globalFolderPath,'') else: remotePath = remoteLocation remotePath = remotePath.replace('\\','/') url = host +'/app/explore/!/upload?vId=0&rename=0&to=/'+remotePath files = [] header = {'Authorization': token} notSentFiles = [] for filename in filesNames: filePath = '%s\\%s'%(folderPath, filename) if compareModificationTime(filePath): files.append( ("file", ( open(filePath, "rb")))) lastRunTime [filePath] = int(time.time()) else: notSentFiles.append(filePath) if files: r = requests.post(url, files=files, headers=header) for filename in filesNames: print(remotePath+'/'+filename, r.status_code, r.reason) return notSentFiles try: f = open(lastRunFilename, "r") timestamp = f.read() lastRunTime = json.loads(timestamp) except: lastRunTime = {} notSentFiles=[] if ( os.path.isdir(path)): for root, subdirs, files in os.walk(path): if files: notSentFiles += uploadFile (files, root) time.sleep(0.1) elif ( os.path.isfile(path)): notSentFiles += uploadFile ([os.path.basename(path)], path.rsplit('\\', 1)[0]) elif not os.path.exists(path): print ("Specified path doesn't exist") if notSentFiles: print ('These files were not uploaded because they were not modified:') for file in notSentFiles: print (file) with open(lastRunFilename, 'w') as f: json.dump(lastRunTime, f) Version >= L2021.5 import uuid import requests import io import json import os import time from datetime import datetime import configparser config = configparser.ConfigParser() config.read('config.ini') lastRunFilename = config.get('DEFAULT','lastRunFilename') host = config.get('DEFAULT','host') remoteLocation = config.get('DEFAULT','remoteLocation') path = config.get('DEFAULT','path') token = config.get('DEFAULT','token') # Cut slashes at the end of paths if they are exist if path[-1] == '\\': path = path[:-1] if remoteLocation[-1] == '/': remoteLocation = remoteLocation[:-1] # Function to compare stored and file last modified time def compareModificationTime(filePath): if (filePath in lastRunTime and os.path.getmtime(filePath)>int(lastRunTime.get(filePath))) or (filePath not in lastRunTime): return True else: return False # Function to upload file to Droppy via Sisense API def uploadFile (filesNames, folderPath): def get_auth(token): headers = { 'Authorization': token, } x_auth = requests.post(f'{host}/app/explore/api/login', headers=headers) if x_auth.status_code==200: return x_auth.text else: print ('Cannot get x-auth token. Check API token validity') return False # # SERVICE FUNCTION # # Do NOT modify x_auth = get_auth(token) if os.path.isfile(path): globalFolderPath = os.path.dirname(path) else: globalFolderPath = path if folderPath != globalFolderPath: remotePath = remoteLocation + folderPath.replace(globalFolderPath,'') else: remotePath = remoteLocation remotePath = remotePath.replace('\\','/') files = [] header = {'Authorization': token, 'x-auth': x_auth,} notSentFiles = [] for filename in filesNames: filePath = '%s\\%s'%(folderPath, filename) if compareModificationTime(filePath): files.append(filePath) lastRunTime [filePath] = int(time.time()) else: notSentFiles.append(filePath) if files: for file in files: with open(file,'rb') as ff: file=os.path.basename(file) r = requests.post(f'{host}/app/explore/api/resources/{remotePath}/{file}?override=true', headers=header,data=ff) print(remotePath+'/'+file, r.status_code, r.reason) return notSentFiles try: f = open(lastRunFilename, "r") timestamp = f.read() lastRunTime = json.loads(timestamp) except: lastRunTime = {} notSentFiles=[] if ( os.path.isdir(path)): for root, subdirs, files in os.walk(path): if files: notSentFiles += uploadFile (files, root) time.sleep(0.1) elif ( os.path.isfile(path)): notSentFiles += uploadFile ([os.path.basename(path)], path.rsplit('\\', 1)[0]) elif not os.path.exists(path): print ("Specified path doesn't exist") if notSentFiles: print ('These files were not uploaded because they were not modified:') for file in notSentFiles: print (file) with open(lastRunFilename, 'w') as f: json.dump(lastRunTime, f) File Management (Upload, Delete, Download, Rename Etc)