Uploading Files Locally Using Rest (Linux)

[intapiuser]

In Sisense Linux distribution you are able to upload files to the Sisense environment via the file management Interface. This article describes few ways for uploading files programmatically using bash, Powershell and python.

Powershell Script for uploading a file via rest api for Sisense version greater than L2021.5

# ------- Config -------

# Sisense base DNS
# For example http://23.50.45.130:30845
$baseUrl="<Sisense DNS>" 

# Destination location on sisense server - must be accessible via Sisense file management interface
$remoteLocation="/test1/data.csv"

# Local file path 
$filePath="data.csv"

# admin api token
$apiToken="<Admin API Token>"

# Should the new file override existing files with the same name
$shouldOverride="true"

# ------- End Of Config -------

# File Management URL
$uploadPathUrl="/app/explore/api/resources/data"

# Create URL to send file to
$url=$baseUrl+$uploadPathUrl+$remoteLocation + "?override=" + $shouldOverride
#Write-Output $url

# Create authorization bearer header
$authorization="Bearer $apiToken"
#Write-Output $authorization

# In order to add files via the file explorer, one requires also x-auth token. the next part fetchs x-auth using the admin api token
$xauthpath="/app/explore/api/login"
$xauthurl=$baseUrl+$xauthpath
#Write-Output $xauthurl

$headers = @{
    'Authorization' = $authorization
}

# Send request to get xauth
# For https remove -AllowUnencryptedAuthentication
$xauth = Invoke-RestMethod -Uri $xauthurl -Method Post -Headers $headers -AllowUnencryptedAuthentication
#Write-Output $xauth

$headers += @{
    'X-Auth' = $xauth
}

# Send a request to upload a file
# For https remove -AllowUnencryptedAuthentication
$res = Invoke-RestMethod -Uri $url -Method Post -Headers $headers -AllowUnencryptedAuthentication -ContentType 'text/csv' -InFile $filePath

# Print response
Write-Output $res

Bash Script for uploading a file via rest api (using cUrl) for Sisense version greater than L2021.5

#!/bin/bash          

# ------- Config -------

# Sisense base DNS
# For example http://23.50.45.130:30845
baseUrl="<Sisense DNS>"

# Destination location on sisense server - must be accessible via droppy
remoteLocation="/test1/data.csv"

# Local file path 
filePath="data.csv"

# admin api token
apiToken="<Admin API Token>"

# Should the new file override existing files with the same name
shouldOverride="true"

# ------- End Of Config -------

# File Management URL
uploadPathUrl="/app/explore/api/resources/data"

# Create URL to send file to
url=$baseUrl$uploadPathUrl$remoteLocation"?override="$shouldOverride

echo $url

# Create authorization bearer header
authorization="Bearer $apiToken"

# In order to add files via the file explorer, one requires also x-auth token. the next part fetchs x-auth using the admin api token
xauthpath="/app/explore/api/login"
xauthurl=$baseUrl$xauthpath
echo $xauthurl

# Send request to get xauth
xauth=$(curl -H "Authorization: $authorization"  $xauthurl)
echo $xauth

# Send a request to upload a file
res=$(curl -H "Authorization: $authorization" -H "X-Auth: $xauth" -H "Content-Type: text/csv" -d @$filePath $url)

# Print response
echo $res

Power-Shell 1 file upload Example - Version <=L20121.3

The script below will allow the user to upload files programmatically, using REST command. The script itself is written in Powershell.

Change the first 4 parameters

dns - The URL you use for your Sisense site

RemoteLocation - The location of the target file within File Manager

FilePath - The local location of the file from the Local Server

AUTH_TOEN - Generate a token and replace it with <token>

$dns= 'https://myhost.sisense.com'
$RemoteLocation= 'data/Test'
$FilePath = 'C:\myfolder\myfile.csv';
$AUTH_TOKEN = 'Bearer <Token>'

###############################################################
$pos = $FilePath.LastIndexOf("\")
$URL = $dns+'/app/explore/!/upload?vId=0&rename=0&to=/'+$RemoteLocation;
$filename = $FilePath.Substring($pos+1)
$fileBytes = [System.IO.File]::ReadAllBytes($FilePath);
$fileEnc = [System.Text.Encoding]::GetEncoding('UTF-8').GetString($fileBytes);
$boundary = [System.Guid]::NewGuid().ToString(); 
$LF = "`r`n";
$Headers = @{'Authorization'=$AUTH_TOKEN};

$bodyLines = ( 
 "--$boundary",
 "Content-Disposition: form-data; name=`"filename`"; filename=`"$filename`"",
 "Content-Type: application/octet-stream$LF",
 $fileEnc,
 "--$boundary--$LF" 
) -join $LF

Invoke-RestMethod -Uri $URL -Method Post -ContentType "multipart/form-data; boundary=`"$boundary`"" -headers $Headers -Body $bodyLines

Python Example For Folder Syncing (Upload Only)

In this example you can set a local folder with sub-folders to sync with a File Management virtual library.

First run will upload and update all files and create a lastRunTime.json file for reference of last run.

On the next run only files that the update date is larger than the last run time of the application will be uploaded/updated

Config.ini file sample:

[DEFAULT]

# Sisense URL
host = https://test.sisense.com

# Path to store files on target machine
remoteLocation= data/Test

# Path to folder or file which should be transferred 
# If folder is specifeed, all subfolders will be also proccessed
path = C:\Users\Documents\Test\

# Sisense API token
token = Bearer TOKEN


# Script will store last modified time for each processed file. This will allow it to upload only modified files on next executions
lastRunFilename = lastRunTime.json

Droppy.py file sample:

Droppy.py example Version <=L2021.3

import uuid

import requests

import io

import json

import os

import time

from datetime import datetime

import configparser

config = configparser.ConfigParser()

config.read('config.ini')

lastRunFilename = config.get('DEFAULT','lastRunFilename')

host = config.get('DEFAULT','host')

remoteLocation = config.get('DEFAULT','remoteLocation')

path = config.get('DEFAULT','path')

token = config.get('DEFAULT','token')

# Cut slashes at the end of paths if they are exist

if path[-1] == '\\':

 path = path[:-1]

if remoteLocation[-1] == '/':

 remoteLocation = remoteLocation[:-1]

# Function to compare stored and file last modified time

def compareModificationTime(filePath):

 if (filePath in lastRunTime and os.path.getmtime(filePath)>int(lastRunTime.get(filePath))) or (filePath not in lastRunTime):

  return True

 else:

  return False

# Function to upload file to Droppy via Sisense API

def uploadFile (filesNames, folderPath):

 if os.path.isfile(path):

  globalFolderPath = os.path.dirname(path)

 else:

  globalFolderPath = path

 if folderPath != globalFolderPath:

  remotePath = remoteLocation + folderPath.replace(globalFolderPath,'')

 else:

  remotePath = remoteLocation

 remotePath = remotePath.replace('\\','/')

 url = host +'/app/explore/!/upload?vId=0&rename=0&to=/'+remotePath

 files = []

 header = {'Authorization': token}

 notSentFiles = []

 for filename in filesNames:

  filePath = '%s\\%s'%(folderPath, filename)

  if compareModificationTime(filePath):

   files.append( ("file", ( open(filePath, "rb"))))

   lastRunTime [filePath] = int(time.time())

  else:

   notSentFiles.append(filePath)

 if files:

  r = requests.post(url, files=files, headers=header)

  for filename in filesNames:

   print(remotePath+'/'+filename, r.status_code, r.reason)

 return notSentFiles

try:

 f = open(lastRunFilename, "r")

 timestamp = f.read()

 lastRunTime = json.loads(timestamp)

except:

 lastRunTime = {}

notSentFiles=[]

if ( os.path.isdir(path)):

 for root, subdirs, files in os.walk(path):

  if files:

   notSentFiles += uploadFile (files, root)

   time.sleep(0.1)

elif ( os.path.isfile(path)):

 notSentFiles += uploadFile ([os.path.basename(path)], path.rsplit('\\', 1)[0])

elif not os.path.exists(path):

 print ("Specified path doesn't exist")

if notSentFiles:

 print ('These files were not uploaded because they were not modified:')

 for file in notSentFiles:

  print (file)

with open(lastRunFilename, 'w') as f:

    json.dump(lastRunTime, f)

Version >= L2021.5

import uuid
import requests
import io
import json
import os
import time
from datetime import datetime
import configparser
​
​
config = configparser.ConfigParser()
config.read('config.ini')
​
lastRunFilename = config.get('DEFAULT','lastRunFilename')
host = config.get('DEFAULT','host')
remoteLocation = config.get('DEFAULT','remoteLocation')
path = config.get('DEFAULT','path')
token = config.get('DEFAULT','token')
​
# Cut slashes at the end of paths if they are exist
if path[-1] == '\\':
 path = path[:-1]
if remoteLocation[-1] == '/':
 remoteLocation = remoteLocation[:-1]
​
# Function to compare stored and file last modified time
def compareModificationTime(filePath):
​
 if (filePath in lastRunTime and os.path.getmtime(filePath)>int(lastRunTime.get(filePath))) or (filePath not in lastRunTime):
  return True
 else:
  return False
​
# Function to upload file to Droppy via Sisense API
def uploadFile (filesNames, folderPath):
 def get_auth(token):
  headers = {
  'Authorization': token,
  }
​
​
  x_auth = requests.post(f'{host}/app/explore/api/login', headers=headers)
  if x_auth.status_code==200:
   return x_auth.text
  else:
   print ('Cannot get x-auth token. Check API token validity')
   return False
  
 # # SERVICE FUNCTION
 # # Do NOT modify
 x_auth = get_auth(token)
 if os.path.isfile(path):
  globalFolderPath = os.path.dirname(path)
 else:
  globalFolderPath = path
​
 if folderPath != globalFolderPath:
  remotePath = remoteLocation + folderPath.replace(globalFolderPath,'')
 else:
  remotePath = remoteLocation
​
 remotePath = remotePath.replace('\\','/')
​
 files = []
 header = {'Authorization': token, 'x-auth': x_auth,}
 notSentFiles = []
 for filename in filesNames:
  filePath = '%s\\%s'%(folderPath, filename)
  if compareModificationTime(filePath):
   files.append(filePath)
   lastRunTime [filePath] = int(time.time())
  else:
   notSentFiles.append(filePath)
 if files:
  for file in files:
   with open(file,'rb') as ff:
    file=os.path.basename(file)
    r = requests.post(f'{host}/app/explore/api/resources/{remotePath}/{file}?override=true', headers=header,data=ff)
    print(remotePath+'/'+file, r.status_code, r.reason)
​
 return notSentFiles
​
​
try:
 f = open(lastRunFilename, "r")
 timestamp = f.read()
 lastRunTime = json.loads(timestamp)
except:
 lastRunTime = {}
​
​
​
notSentFiles=[]
if ( os.path.isdir(path)):
​
 for root, subdirs, files in os.walk(path):
  if files:
​
   notSentFiles += uploadFile (files, root)
​
   time.sleep(0.1)
elif ( os.path.isfile(path)):
​
 notSentFiles += uploadFile ([os.path.basename(path)], path.rsplit('\\', 1)[0])
​
 
elif not os.path.exists(path):
 print ("Specified path doesn't exist")
​
if notSentFiles:
 print ('These files were not uploaded because they were not modified:')
 for file in notSentFiles:
  print (file)
​
​
with open(lastRunFilename, 'w') as f:
    json.dump(lastRunTime, f)

File Management (Upload, Delete, Download, Rename Etc)