has joined #papers06-ai-digital-platforms-web-based

@joe.wright set the channel purpose: Paper Session 6: AI / Digital Platforms / Web-Based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

@joe.wright has renamed the channel from ?papers6-ai-digital-platforms-web-based? to ?papers06-ai-digital-platforms-web-based?

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

has joined #papers06-ai-digital-platforms-web-based

Papers for this session: ? *Support System for Improvisational Ensemble Based on Long Short-term Memory Using Smartphone Sensor* (Paper 77) H. Takase and S. Shiramatsu ? *On Digital Platforms and AI for Music in the UK and China* (Paper 69) N. Bryan-Kinns, Z. Li and X. Sun ? *A Survey on The Uptake of Music AI Software* (Paper 95) S. Knotts and N. Collins ? *Sonification of High Energy Physics Data Using Live Coding and Web Based Interfaces* (Paper 76) K. Vasilakos, S. Wilson, T. McCauley, M.K. Mardakheh, E. Margetson and T.W. Yeung ? *Patch-corde: An Expressive Patch Cable for The Modular Synthesizer* (Paper 120) J. Wilbert, D. Haddad, H. Ishii and J.A. Paradiso

Immediately following this session will be Lunch and the OHMI Dropin: https://nime2020.bcu.ac.uk/ohmi/

** Live in 4 minutes with paper session 6 - see you there!

Zoom: https://us04web.zoom.us/j/74581865900?pwd=eHRHcjVNaitkcWM1cDFZc1UrWW1PQT09

Just in case we have issues with Zoom Captions, this is the link for the external captions: https://www.streamtext.net/player?event=NIME230720

we're live now!

https://www.youtube.com/watch?v=M2aXVLSY9ko&feature=youtu.be This week I actually uploaded a demo movie of the system at this stage to You Tube. I post the URL on slack, so please take a look!

Could you discuss the LSTM model in more detail? Perhaps the model architecture and relevant parameterisation?

Hi @h.takase thanks for the talk really interesting - modern smartphones often incorporate barometers I wonder if you have considered using that for estimating vertical position (altitude)?

What is the application of the Kinect in this performance system? Does it have the same function as the smartphone?

Interesting graph of the two axis of rotation for detecting attack. Do you think that has applications for other gyroscopic/accelerometer device mapping?

could be wrong but looks like it's ugmenting the phone data with a height value?

Second paper presentation starts now!

And please mention the author's name when asking a question, for those who trace back the conversation later on! So we are now at @n.bryan-kinns

Is @jason.hockman muted?

Very interesting @n.bryan-kinns - do you have any ideas for follow-ups on this study?

I use Kinect to collect teacher data for Vertical motion estimation. The Kinect records the height of the hand holding the phone. The vertical motion estimation LSTM model estimates the height of the hand holding the phone. The sum of the actual values between attack timings is used to determine vertical movement.

@n.bryan-kinns could you elaborate on adaptive music techniques? what's their purpose and mechanics?

And: in relation to other discussions we have had about a Western bias of a lot of NIMEs, it would be super-interesting to apply a similar approach to what you have done also to other countries. @n.bryan-kinns

Sorry all, my Zoom crashed.

I hadn't considered using a barometric sensor. We'll use your message to help us with future research. Thank you?

Very interesting, wondering is there a way to quantize the cultural aspects behind Chinese music in terms of performance tech etc?

@n.bryan-kinns so the music would be fundamentally synthetic?

super interesting, thank you!

next up @knotts.shelly !

@knotts.shelly interesting that you found that participants did not think that AI would take away their jobs, and are more seen as an opportunity - we found the same thing in our study in the UK and China (not reported in our paper). But, our participant sample was self-selecting and had a strong interest in digital music and AI anyway.

@knotts.shelly what's your own take on the date question - i.e. when AI might attain human-level music making?

@quinnjarvisholland it's possible to use other method. Could you teach me the names of the other methods for gyroscopic/accelerometer device mapping?

@knotts.shelly Really great talk!

@knotts.shelly very interesting, thanks for doing this work! Such a nice counterbalance to the overhyped and oversimplified narratives we see getting so much attention in the broader world.

@knotts.shelly Thanks for the very interesting work! :slightly_smiling_face: I might have missed it at the beginning of your talk, but what definition of « music AI » were you using in the survey? Are we specifically discussing machine learning models for mapping or music generation? Or do we include systems with autonomous decision-making abilities, such as shallow classifiers, or even rule-based software (which might be put into AI to some extent)?

Nice job @knotts.shelly! But maybe the mostly positive attitude has to do with self-selection of participants? Those that decided to go ahead and fill the survey are probably somehow more attracted to music AI than those than did not?

Thank you for an interesting presentation @knotts.shelly! :smile: We published a paper focusing on AI tools for making music in the context of video creation at CHI this year: https://dl.acm.org/doi/abs/10.1145/3313831.3376514 Some of the questions that you asked in your survey were similar to the ones that we asked in our surveys :slightly_smiling_face:

@knotts.shelly It's not a silly question and thanks for a great answer!

oh great to know. i'll check it out!

@jason.hockman LSTM is suitable for estimating time series data such as music and text data because having a layer that retains past output. Actually LSTM gives better accuracy in attack timing estimation and vertical motion estimation than Bayesian network in this result. I hope that answers your question.

Perhaps the question is when computers will start musicking by themselves, and only then we can consider it AI creativity. As long as it is humans functionally applying AI applications in their musicking pursuits, the creativity is still with the human.

we didnt define AI at all - for the reasons i just explained. The software use section listed a number of softwares that either claim to use AI (commercial softwares) or are programming tools with AI libraries or potential.

Fourth presentation is running now! @konstantinos.vasilako

@marije Shouldn't we be putting the question to the AI systems? That might really tell us how smart they are!

@knotts.shelly thanks for your answer :pray:

yes, that's an excellent point. actually the wording was "independent musical AI" so that was the implication.

thank you!

a trailer of this project can be found also here https://www.youtube.com/watch?v=U2aDudtCiY4

the positive vs. negative responses was fairly evenly split. there were lots of people who made fairly grumpy comments about the state of AI in music making in any other comments so i dont think people necessarily de-selected due to not liking AI.

That?s true, somebody anti-AI could have as easily decided to take the survey!

?copy-pasting my message here?thanks for your answer!! :)

I don't have any names for methods really. Just have used mapping "pitch" and "yaw" or x and z rotation to pitch as well as filter cutoff on a synthesizer. https://github.com/pccadaptiveinstrumentsteam/PCC-Adaptive-Instruments-Project my team project here <--- I like the inclusion of the kinect- we also could not get a gould altitude reading without augmenting it with something like that

yes i think we comment briefly in the discussion section of the paper on the need for AIs that can engage in creative dialogue :slightly_smiling_face:

@konstantinos.vasilako Interesting! since the data is not coming in realtime (I guess?) I am wondering wether the interface gives some insights into the time dynamics of the data, as that might influence the mapping choices

ok, so it is about sonifying each event, and then aurally distinguish different events by the mappings chosen.

Hi @konstantinos.vasilako really interesting project - I really like the idea of doing live sonification design and live coding is a good way to do this. In the work I?ve done with scientists there is sometimes some skepticism over the merit of sonification. How did the scientists at CERN respond to this work? Can they use it did they like it?

Yes, basically @marije. There?s an idea of an active collision event, which can be swapped while sonifying. So you can compare different events with the same mapping.

Next paper is starting! Last one of the session.

@tom.mitchell They were firstly very happy that we were using the data, as a lot of art projects there are more ?in response to? the physics than data driven. Maurizio was very happy and said that we ?got it? which was very gratifying. We did discuss ways in which it might be usable for them, but we haven?t yet got to that.

@tom.mitchell the responses we got so far, is that it helps them to think about this data in a little bit out of the box of the way they are used to approach it, so yes I can say we got positive responses.

ok, cool!

this is rad!

This is such a great idea!

*QUESTION*: Is it necessary to have a decoder module? What happens when you use the patch cable 'as-is' between different sorts of modules?

QUESTION: I missed have missed it: did you mention whether or not there is hysteresis?

QUESTION: How durable is the cable? Does the conductive material maintain the same properties in the long term?

in my experience they tend to rip quite easily. We discussed the same question yesterday as well: https://nime2020.slack.com/archives/C017H5QEY3W/p1595428289086200

could it be confusing perhaps with all the other wires in performance to pull on and manipulate the wrong one, could the cable be made slightly differently also different colour to the other ones?

Got to do it right, can imagine yanking on one too hard and breaking somethingXD

thank you all!

For those who run off now: check the installations! https://nime2020.bcu.ac.uk/installations/

expressive controller! Patch-cord!

OHMI drop in right now ! https://us04web.zoom.us/j/77782338161?pwd=c0Y0ZW9XcTYvNzB1VDJUMnAzNHBUUT09

@konstantinos.vasilako - hi, I really liked you project, and it also reminded me of a project I presented in 2018. https://www.youtube.com/watch?v=JUSO-6ykuIo - it was based on sonification of the data from CERN particle collision as well,. We also scaled the data arbitrarily to various ranges that later controlled all parameters (freq, del. time, amount of feedback, ?). From what I see here is that you also converted the data rather freely so that it suits artistic needs which is great. My only criticism to myself was that I wish I had found some underlying connection, because, even though the data was taken from the particle collision, it could have (hypothetically) been a series of random numbers. My question is, why was this particular collission data important to you, did you notice some underlying connection, or what have you done in order to validate the use of this data as opposed to some random numbers in the process of sonification. THANKS :slightly_smiling_face:

Thanks for your patience with technical issues, we have limited capacity to run over as our hard-working captioners need time to rest in between the discussions. Please keep your questions coming. And thanks to all the presenters for another really interesting session!

Thank you captioners! :trophy:

@vincze I?m sure @konstantinos.vasilako will have much to add as well, but it?s a general problem with sonification, often at odds with the conceit that you?re hearing the ?sound? of something that media and public seem so keen to endorse. That said it is of course possible to have good and bad sonifications, just as you can have good and bad visualisations. We mean this in the sense that the sonification brings out or clarifies salient aspects of the data and its character. To the extent that we succeed at this the result is in some sense meaningful, and not random. Hope that makes sense.

Your project looks very interesting, btw!

I guess the need for the extra electronics is to allow for the use of the full voltage range modules expect/support [fix impedance and source signal level]. And probably the dual supply configuration was chosen to use this tech to modulate audio signals running between modules too, rather than only CVs. But I am looking ff to hearing from the authors!

@vincze Yeah I see your point that it could be also a random/chaotic noise generator doing some live feeding, however what we noticed after several performances, was that the data was carrying some noticeable patterns between each events, which allowed one to map it to specific parameters apropo to their control signal e.g., some event was working nicely for Freq, or for modFreq etc, while other events were providing a different character. On top of that, what was important for the collectiveness in the ensemble and for the creation of coherence, was that all are using same input of raw data and thus this works as a starting point for everyone.

@quinnjarvisholland We also made a system that allows you to change the pitch at an angle of your smartphone and produce sound by touch. However, it is difficult to simply calculate the vertical motion of the smartphone while shaking it, as we did in this study, so we tried to improve the accuracy of the vertical motion estimation by training LSTM using the vertical motion of the smartphone obtained from Kinect as teacher data.

@s.d.wilson and @konstantinos.vasilako - thank you guys for your answers, as Scott pointed out, it is a general question when doing this kind of sonifications/visualisations what one is to do with abstract data, especially when it is complex. One can always assume that that must be some patterns if the data (as in your case) is not random, but I guess the trick is to notice that and have it work in your favour. Really like your work, generally very curious about these kinds of projects that bind particle physics with art/sound, because the physical models are often so abstract and crazy that it borders with art practice anyway. For us, the unifying element was that 15m long string that was symbolising the idea of the string theory, while data was driving the whole installation. Cool getting to know your work, wish I could have been there. Any future performances planned?

@vincze Thanks! We are performing DarkMatter on Saturday.

@h.takase Many thanks for your reply. Are you using the cell independently or within a layer of a network (e.g., RNN)?

@konstantinos.vasilako - oh great, as a part of NIME, virtually? What time?

@vincze Yeah we will attempt to make a telematic version of it this Saturday here at *22:00-22:50* (UTC + 3) *session.*

(UTC + 3)

The live streaming set is here: https://youtu.be/4C8E559Pc30

Also have a short prerecorded version in Music 4 tomorrow: https://www.youtube.com/watch?v=m0A3C1imhhw&feature=youtu.be

@konstantinos.vasilako & @s.d.wilson / Thank you guys, good luck with the performance, looking forward!

Cheers!

Very cool - I?ll stay tuned?

@jason.hockman We are using simpleLSTM from keras. Since LSTM is an extension of RNN, the memory cells are contained in the network layer. The difference between RNN and LSTM is that RNN simply keeps data up to the present, whereas LSTM has a forget gate to select the past data to keep. RNN causes gradient loss by using all the past data for the current training, but LSTM avoids this problem thanks to the forget gate.

Oh, thank you so much! Your performance is tonight right?