Rendered at 07:21:07 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
maxgee 2 days ago [-]
Our account was recently banned from OpenAI for a CSAM violation. We had an autonomous agent using an image classifier switch to OpenAI (instead of a local model it should have been using) which was classifying user-uploaded content. After classifying over 100,000 images, OpenAI banned our account.
We asked for further information so we could report the uploader to the NCMEC and remove the content from our systems. They refused to give us any information about this incident, denied our appeal, and have so far, illegally refused our privacy requests for data about our account.
OpenAI does not care about fighting child sexual abuse, this is completely performative.
EDIT: These image uploads are up to 20+ years old, and are being served through Cloudflare with their CSAM scanning product enabled. We've received no notification of any problems from Cloudflare. Something on OpenAI's system thinks that an image we sent to them for classification is CSAM, and they aren't willing to elaborate or communicate about it at all. For reference, our website is https://ytmnd.com
uptrails 2 days ago [-]
I've done some work in this space and it's mostly that Big Tech has regulators and parent advocacy groups already very angry with them and breathing down their necks, so they would rather lose customers than take a nuanced approach. The marginal value of having you as a customer is approximately zero, the regulatory risk of letting CSAM through is huge.
Another complicating factor is that it really blows to have humans look at stuff like that, so everyone would rather use automated classifiers; and if you assume good faith and share too much info with the affected parties, this will be exploited by the bad guys to gain insights into how the system works and how to circumvent it.
I don't think there's an elegant solution to this, other than "don't allow tech companies to get so big that they can afford not to care about individual customers".
methodical 1 days ago [-]
I'm unsure if there's any particular size requirement for a company to not care about individual customers. Barring an exceedingly small amount of businesses who rely on a handful of customers for their entire bottom line, probably almost all B2C and most B2B companies have a certain point at which any particular customer is simply not worth the overhead to serve. For OAI I would assume anything that even smells like CSAM is probably more than enough for them to justify acting first and (maybe) asking questions later, especially with the regulators and IPO to consider.
limagnolia 1 days ago [-]
So you agree with the OP that OpenAI doesn't care about CSAM and the post in performative for the advocacy groups you mentioned?
uptrails 1 days ago [-]
I don't think they hire people who care about it any less than the general public. There's plenty of parents in big tech, and that includes execs.
But as a matter of self-preservation, their policy is basically "don't ever bring CSAM to any of our platforms, period". It's the same as Google, Apple, Meta, etc. It doesn't matter if you're a researcher, an investigator, if it's an accident. They will ban you because they don't want to be in the business of adjudicating this stuff and because your account doesn't matter to them.
It's completely orthogonal to caring or not caring about the issue.
limagnolia 17 hours ago [-]
Pretty sure that Google and Microsoft both offer tools specifically designed for handling possible CSAM. But the point was that if OpenAI cared about it, they would be willing to do more to cooperate with their past customer in fighting it. But they don't, so they won't. Maybe the other companies you mentioned also don't care (probably).
nvarsj 12 hours ago [-]
Your solution doesn't make sense. It's much easier to enforce this stuff with centralization.
vintermann 20 hours ago [-]
> The marginal value of having you as a customer is approximately zero, the regulatory risk of letting CSAM through is huge.
I agree, almost. The risk isn't letting CSAM through though, the risk is "being seen as not doing enough". And being a hyper-anal mechanical pharisee is good for being seen as doing something.
deaux 1 days ago [-]
> I don't think there's an elegant solution to this, other than "don't allow tech companies to get so big that they can afford not to care about individual customers".
On the contrary, that's an incredibly elegant solution.
icantevenhold 2 days ago [-]
I would posit almost no one really cares about fighting child sexual abuse; most of them just use this as pretence for more surveillance and data collection.
hingler36 2 days ago [-]
I would posit the exact opposite: it's really important to fight child sex abuse, which is why it's so frustrating that people are co-opting that cause to promote surveillance and data collection.
icantevenhold 1 days ago [-]
Oh you misunderstood me; it’s important for sure.
I don’t think this is something that can be solved by more surveillance though.
Most abuse (i think something like 80%? Not sure but it was a big number) happens within the family and between minors (minors abusing minors)
Changing how easy or hard it is to distribute or generate CSAM will help exactly nothing here even if that’s the goal.
Joker_vD 1 days ago [-]
You know, when you stop to think about it, what does forcing the major file/image hosters to overtly delete CSAM (and ban the poster accounts) exactly do? That's right, it forces the people who create/share/exchange CSAM to go for the secured/encrypted messengers where that content is pretty much impossible to intercept and detect — which obviously makes the task of finding and arresting those people much harder. Like, just make those hosters to tacitly contact the relevant authorities and not tip off the criminals that they've been detected. Nah, let's push those guys to use Signal and TOR.
mminer237 1 days ago [-]
Typically they do report people to the police as well. But once the image is uploaded, they have as much info as they're getting probably. Hosting it for months until police get an indictment doesn't really do any good and just perpetrates these images' spread. I think criminals already know that most mainstream image hosts try to detect CSAM, whether they delete it immediately or not.
estearum 1 days ago [-]
Obviously there's all flavors of coordination with law enforcement to do exactly what you're talking about when it's appropriate.
itake 1 days ago [-]
It also pushes big players (like FB or Apple) to add e2e encryption in their services so they can avoid being blamed for "knowing" that someone was planning an act of violence.
AI unfortunately currently can't be e2e encrypted at scale, so the the big players are forced to use surveillance to protect their legal butts from being sued.
Imagine if the government (USPS) decided they need to open and read every single letter or package you send or receive, for /safety/.
fc417fc802 1 days ago [-]
> AI unfortunately currently can't be e2e encrypted at scale
It can actually, there are even inference providers that offer this. It isn't strictly E2EE but running a published image on attested hardware is the next best thing. The customer can inspect the image and the hardware manufacturer attests that this is the image that the inference provider is actually running.
That said the latest generation of attested hardware is quite vulnerable (see https://tee.fail/) so you're still taking the provider at their word that they haven't intentionally compromised their own machines.
owenmarshall 1 days ago [-]
> Imagine if the government (USPS) decided they need to open and read every single letter or package you send or receive, for /safety/.
Of interest for those who might not know, the USPS does this for _metadata_ today.
The US Postal Service photographs every single article of mail it processes in the US through a program called Mail Isolation Control and Tracking[0]. It was created post-2001 anthrax scare, and is widely shared across LEOs.
But unlike most LEO intercept systems, you can actually /benefit/ from this one and get your own lens into their data feed.[1]
Saying it's important is not the opposite of saying that it's rare to actually care.
demibabs 2 days ago [-]
How else do you prevent it tho?
joe_the_user 1 days ago [-]
One key means of preventing the exploitation of children is placing children in situation where they aren't powerless, where their caregivers aren't in such a marginal position that they can't prevent predators from exploiting them.
Basically, keeping children from being miserably. Poverty isn't the only source of the exploitation of children but it's huge one and how little any anti-CSM crusaders care about this reveals a lot imo about their lack of sincerity.
But in our society, even in a much better society than this, bad things are going to happen at scale. At a certain point, you have abandon the "we'll do anything to stop this terrible thing, no matter how much what you do hurts people otherwise, no matter how ineffective what you do is, you must do your thing since it can be done".
plagiarist 1 days ago [-]
Another great way to prevent child exploitation is age-appropriate sex education for the children. But we don't see politicians promoting that.
graemep 1 days ago [-]
It is complex and controversial. What is appropriate to what age? Who should do it? How much should it be driven by children's questions and how much regardless? Should all children by taught the same things at the same age? Should it be purely biological or do you teach them values at the same time? If so whose?
icantevenhold 1 days ago [-]
It’s not so hard.
> What is appropriate to what age?
Pretty much everything. You can talk to children about sex without issues.
> Who should do it?
A person the child can trust, can be a teacher or parent for example
> How much should it be driven by children's questions and how much regardless?
If they have questions answer them.
> Should all children by taught the same things at the same age?
Yea I don’t see why not
> Should it be purely biological or do you teach them values at the same time?
Both
graemep 21 hours ago [-]
Its not so hard in a world where everyone agrees with you. The problem is that people do not agree. Any policy imposed by politicians will make some people happy, will have a slot of people partially disagreeing with it, and some people strongly disagree with it.
If you look at sibling comments they disagree with you. One assumes a syllabus taught by teachers and says values should not be taught. If you do teach values whose values? There is nothing people disagree about more than sex.
You say "a teacher or a parent" for example. If its a politically imposed system it has to be the former. I think it is better for it to be the latter.
Children are very different so they should not be taught the same thing at the same age. Its a general flaw of classroom education but its even more of a problem with something where maturity matters. I disagree with everything being appropriate at any age - you notice GP said "age appropriate".
We have had as many different views in this thread as people have commented. I think that proves my point.
camgunz 20 hours ago [-]
People believe a lot of dumb things. It's not an excuse to put an extremely low ceiling on public policy. If you don't teach your kid about body safety and autonomy (an extremely easy thing to do as long as you're willing to say words like "vagina", "vulva", "penis", etc) it's the same as not teaching them the stove is hot.
graemep 18 hours ago [-]
> People believe a lot of dumb things. It's not an excuse to put an extremely low ceiling on public policy
It is in a democracy
> If you don't teach your kid about body safety and autonomy
I agree, but even people who agree will disagree vehemently about the details when you try and turn it into policy. Lots of people will want to add stuff. Lots of people will want to remove particular things. lots of people will want things taught differently or a a different age.
camgunz 15 hours ago [-]
>> People believe a lot of dumb things. It's not an excuse to put an extremely low ceiling on public policy
> It is in a democracy
No, no man. We don't say, "look, most of us think that enslaving Group X is morally bad, but hey some people will disagree, so we're regrettably prevented from forbidding it." On the other hand, we do say, in several states, "listen, you may be a doctor who thinks abortion is a human right, but we absolutely require under the threat of delicensure and imprisonment that you try and talk women out of them."
People will be as nuts as you let them. There's no reason to tolerate this extreme stuff, especially to the detriment of us all. What's that meme, men will do anything to avoid going to therapy, the internet will require age verification DNA scans to avoid telling their kids about sex.
jampekka 1 days ago [-]
A lot of countries seem to manage just fine.
graemep 21 hours ago [-]
Do they? A lot of countries have sex education in schools but here in the UK the quality of delivery is so bad that I have heard of kids leaving school not having been told about menstruation (the girls will know, obviously). Universities need to run classes about issues like consent.
whattheheckheck 1 days ago [-]
No more of that, people are made Brave New World style in batches of clones with various oxygen levels.
In veil of ignorance Rawlsian style you teach them enough where if you were in their position you wouldn't feel slighted by the adults setting you up for the world at large.
LorenPechtel 1 days ago [-]
Society should look at children's questions. Plot what's asked at what age. The common ones become part of sex education. The others, answer whatever's asked. Hiding the answers never improves the situation.
Keep values out of any mass education.
1 days ago [-]
win311fwg 1 days ago [-]
It may be important, but remains that nobody really cares.
DrammBA 1 days ago [-]
I'm not sure your definitions of "posit" and "exact opposite" align with common usage.
1 days ago [-]
crystaln 1 days ago [-]
That’s dark. For sure it’s used as a pretense for surveillance, however that is not mutually exclusive with caring about child sex abuse.
joe_the_user 1 days ago [-]
I'd agree but I'd add a lot of things are done to get the child sex crusaders off a given corporation's back. Moreover, the crusaders will use every victory to widen their definition of bad things (which isn't me saying anything "good" is here) and number of areas they can trawl for this bad stuff. Enlarged enough, the agenda essentially would include criminalizing pornography and any "deviant" sexuality.
But this agenda is also ultimately about the power of those pushing it rather than just its goals, of course.
grvdrm 2 days ago [-]
I agree. Because if they do they hold some sort of view on what is or isn’t in an image/video. I think it is simpler and easier to put your hands over your eyes and look the other way.
kikdij 1 days ago [-]
Reminds me of a newsroom (the AFP) who was constantly publishing on their social media account conspiracy theory debunking. I told them it was a problem, because they were actually exposing me to those theories: I would not have heard of them without reading the AFP feed, and now I would remember something about it months or years later, maybe not remembering clearly if it was true or not. I recommended than instead of posting their debunking in their main feed, they should post it in discussions on the debunked subject. They never answered and never stopped. And I understood why: they don't want to stop conspiracy theories. They want to be seen fighting them. It's all posturing as well.
rafram 1 days ago [-]
I... am not surprised that the AFP didn't respond to, nor act on, your tweet.
kikdij 21 hours ago [-]
That's what assumptions usually do.
I didn't tweet, I wrote them an articulated email exposing the issue.
GolfPopper 1 days ago [-]
To be fair, they're also trying to use it as PR to polish their rapidly tarnishing corporate image.
nonethewiser 2 days ago [-]
[flagged]
graemep 2 days ago [-]
I think the good faith interpretation of GP's comment is that almost no one among big tech companies (and maybe governments) cares.
nonethewiser 2 days ago [-]
I think the good faith interpretation of his comment is that most people and companies dont care, which Im responding to. If the commenter doesn’t agree with his own words he should clarify.
I think virtually every person and company does care. In fact I think its exceedingly obvious.
Even if OpenAI, Facebook, etc dont care be cause they are evil, they do care because pf liability.
None of this even excludes the possibility that CSAM controls are co-opted for power sometimes. Which is why it’s absurd to suggest these companies don’t care - you can concede that point and still argue its being co opted.
zelphirkalt 1 days ago [-]
So they don't actually care about CSAM existing, they only care about not losing money due to it being on their platforms. Did I get that right? In that case you are confirming what icantevenhold is expressing.
To actually care about something is not the same as doing arbitrary actions due to financial reasons. That's caring about money, and nothing else.
nonethewiser 1 days ago [-]
Im not confirming what icantevenhold said because what he said is broader than the narrow point of liability=caring. Executing on operational requirements to remove CSAM looks a lot more like caring than actively fearmongering people about CSAM to justify new powers.
What do you think? Do you think Facebook, OpenAI, etc. are fine with child porn on their platforms save for liability? A liability which of course exists because virtually everyone agrees that it's a real problem and not just an excuse to grab power.
zelphirkalt 1 days ago [-]
Personally, I think they are fine with it. If there was no law against it, and if they benefited even 1 cent from it, they would be all for it. These tech giants are so profit-oriented, they don't care one bit, who has to suffer for it.
nonethewiser 1 days ago [-]
I imagine 99% of employees, executives, board members, etc. are deeply offended by child porn. You think they just put all those concerns aside for 1 penny? or that the percentage is significantly lower?
Jimmc414 1 days ago [-]
I had the same experience with OAI. We were all-in with OpenAI as our primary provider. We used it exclusively for python and go programming requests. Received a ban for "inciting violence". Appeal denied, no refund, no human response or explanation, 3 years of data lost (privacy requests for data denied). We're on team Anthropic now. Still stings when I think about it.
techpression 1 days ago [-]
Anthropic acts the same way (from experience), so take the learning of backing up your data from this, if nothing else.
randbyte 1 days ago [-]
So just go from bad to worse? I thought the lesson should be trust neither and pick an open weight provider from the dozens?
Jimmc414 1 days ago [-]
Perhaps. But we aren't even riding the line of potentially ban-able usage. There is still that option if it happens again, but right now the benefits outweigh the risks. We were left with few options.
randbyte 11 hours ago [-]
Were you “riding the line of potentially ban-able usage” with OAI though?
Jimmc414 10 hours ago [-]
Absolutely not. No reverse engineering, no license infringement, nothing that could remotely resemble “inciting violence”. We are working exclusively on an accounting workflow based SaaS for law firms.
randbyte 7 hours ago [-]
Yeah then that’s the risk. The compliance flagging is completely opaque and error-prone. And there is no recourse. You are now just betting Anthropic won’t flag you as non-compliant for reason only they know.
lossyalgo 1 days ago [-]
What happens if Anthropic bans you? Where will you go? Perplexity? Do you have a contingency plan in place?
mrheosuper 1 days ago [-]
> We're on team Anthropic now
What makes you think same thing won't happen again ?
mattmaroon 2 days ago [-]
I’m sorry that happened to you, but I would read this more as they are an organization not prepared for this. Why would they be? One minute you’re trying to make computers pass the turning test, the next minute child porn is getting involved.
The fact that they did not previously have proper procedures in place for something like this is not proof that they don’t care. They are one of the fastest growing companies of all time, it’s hard to get disparate parts of an organization aligned on something. Their organization has more than doubled in size in the last nine months.
zelphirkalt 1 days ago [-]
Though of course fastest growing is not any excuse to not be prepared. Maybe the growing should happen in other directions too, than only investor money and evaluation. There are other axes of growth than the financial one.
unpleasurable 1 days ago [-]
uh actually it's good that they're bad and they're only bad because they're so good, cut them some slack will you it's not easy being rich
nonethewiser 2 days ago [-]
> OpenAI does not care about fighting child sexual abuse, this is completely performative.
That doesn’t really follow from your experience.
OpenAI could care about child sexual abuse and not respond to your appeal, even wrongfully.
ezfe 2 days ago [-]
But part of that was trying to identify which content was flagged so that the uploader could be identified, but since it sounds like they didn't identify the content now the OP cannot report it to the authorities.
rpdillon 1 days ago [-]
OpenAI was given a chance to help someone fight child sexual abuse. They declined to do so, potentially violating the law in the process. It's a pretty straight line, given the facts of the post.
maxgee 2 days ago [-]
If they were serious about stopping CSAM they'd spend 10 seconds looking at the prompt we used, the emails we've sent and the fact we're trying to get an NCMEC case number and report the uploader, but we can't because we literally don't know what image they're referencing. If a human was involved in any part of this process or decision, it isn't apparent because all we've received is automated responses.
Frankly I am not interested in retaining an OpenAI account anymore so the only reason we appealed was to try to get more information, not to restore the account.
ajb 1 days ago [-]
Because CSAM is illegal to transmit in many jurisdictions, and sending a link to it is considered transmitting, it's quite possible that they don't think they can legally inform you. The presumption being, if NCMEC think it's CSAM, you will find out then, when they ask you to remove it.
maxgee 1 days ago [-]
They informed us our account was being deactivated for CSAM. We were seeking literally any metadata to help us track it down (a batch number, a content hash, a description, a date and time it happened so we could narrow down the search by looking in our logs, etc), and they have refused to provide any information to us, at all, despite our eagerness to quarantine the asset(s) and report the person who uploaded it to the proper authorities.
At the very least, they could provide us with an NCMEC case number so we can proactively work with the NCMEC to track the content down, but it appears their main motivation is not "strengthening the fight against child sexual abuse" but rather "reducing OpenAI's liability in the fight against child sexual abuse".
fc417fc802 1 days ago [-]
Transmitting to a third party is not at all the same as reporting a url (or hash or what have you) directly to the host. Consider the real world analogy of informing a property owner that you noticed something illegal in his backyard versus telling other unrelated parties where they can obtain the illegal thing.
ajb 1 days ago [-]
Unfortunately, it doesn't have to make sense to become law.
fc417fc802 1 days ago [-]
Fortunately, we (at least in the US) have a common law system and judges that aren't overly literal robots (despite popular misconceptions to the contrary). Regardless of the precise wording of the law you aren't going to get in trouble for notifying a service operator about a url that they themselves host.
If you doubt this, consider that the "report" button on any website does pretty much exactly the same thing at the end of the day.
ponector 23 hours ago [-]
They can set up an AI agent for this. It's not illegal if done by agent, apparently.
charcircuit 2 days ago [-]
His point is that even if they do care, they are being performative in going completely silent, doing "nothing" as opposed to working together with others to remove content and report bad actors.
kkukshtel 1 days ago [-]
I run a moodboard website (sort of like pinterest, but no social features) and I have been worrying about this as well. The site is picking up pretty decent traffic and has hit nearly 1TB in uploaded images, but I haven't yet started scanning incoming images for CSAM content.
Are there any resources or tools you'd recommend here for me to setup? It's already going through cloudflare, and I'm actually in the process of shifting my transform service from Supabase to a hosted container there, so I've got my images routing through there anyways (but storage on Supabase still instead of R2 for now).
throwaway81523 1 days ago [-]
If OpenAI actually flagged something as CSAM, they were supposed to report it to NCMEC. I wonder whether they will confirm having done so, even if they won't say which image it was.
I've so far managed to resist using OpenAI and similar for anything, except having played around with duck.ai a little bit. I'd only want to use local models, preferably models that I trained myself. "Open weight" seems like perverse terminology, like "open binary" for closed source software.
trollbridge 1 days ago [-]
Thanks for running ytmnd. When I was a child, it brought me a great deal of (juvenile) amusement.
Duck roll…
delichon 1 days ago [-]
Classifying unreviewed user-uploaded content with a service that does not accept CSAM is a footgun, even if they misclassified something this time.
harshreality 1 days ago [-]
Imagine you have a small forum that can run cheaply on one server/k3s cluster. You want to classify new content to detect most spam and ease administrative burdens. You rate limit classification[1] so you won't be hit with surprise bills. Sounds fine? Nope, because now any idiot can submit CSAM and have your AI provider drop you.
So basically, table stakes for running a user-submitted-content site will be to pay for your own GPU node/cluster and manage your own inference stack (running a worse model than you could use in the cloud). Otherwise you're at the mercy of opaque and kafkaesque (by necessity) service termination without warning for CSAM (real or hallucinated), and subsequent refusal to communicate about anything related to it.
[1] either by queuing all posts for classification before showing them publically, or just best-effort after-the-fact detection of spam or abuse, and upon detection they're moved to a spam queue.
fc417fc802 1 days ago [-]
> So basically, table stakes for running a user-submitted-content site will be to pay for your own GPU node/cluster and manage your own inference stack (running a worse model than you could use in the cloud).
Or just use one of the many third party inference providers that markets an image classification endpoint intended for user generated content.
nwlieb 1 days ago [-]
What is the price differential between those providers and the frontier labs, on the capability/cost curve? Last I checked, the price difference is on the order of 4-5x for equivalent Artificial Analysis scores.
fc417fc802 1 days ago [-]
Why does the artificial analysis score matter? Aren't we just talking about rapidly (and cheaply) screening user content to prevent the most blatant abuse? "Reasonable" classification performance from open models can be readily had from inference providers at dirt cheap prices.
SamBam 1 days ago [-]
> We had an autonomous agent using an image classifier switch to OpenAI (instead of a local model it should have been using)
The fact that people have autonomous agents making these kinds of decisions is still mind-boggling to me. Maybe I'm stuck in the past. But could this agent have been racking up things of dollars in charges because of this decision GP didn't even know about?
1 days ago [-]
4ebyrawka00 1 days ago [-]
[dead]
randbyte 1 days ago [-]
Performative and just try to shed liability.
chinathrow 1 days ago [-]
If you throw huge amount of work loads at an LLM and expect 100% correct work in order not to loose your account, you might want to reconsider your expectation, no?
cyanydeez 1 days ago [-]
it's not performative if it's done to reduce or eliminate liability. Performative is about signally for the purpose of signalling. Businesses dont signal for it's own sake. They do it to either increase business or decrease liability.
pluc 1 days ago [-]
Performative would be giving the people (everyone) the means to expand the crime they seek to punish
sandboxdevelope 1 days ago [-]
the biggest shocker of this comment is that YTMND is still going strong
threecheese 1 days ago [-]
Hope it wasn't Batman-is-a-dick huwehweheheheheheh, ahh the memories!
victor22 1 days ago [-]
Hugs from Brazil. the original brainrot. uss enterprise etc
snaccident 22 hours ago [-]
To be fair any company can choose who they do business with. And I might have made the same call looking at your… “website”.
NooneAtAll5 1 days ago [-]
why even have openai account then?
stay away, use deepseek and stuff
surgical_fire 1 days ago [-]
Oh my, ytmnd!
I used to view shit there a lot sometime in the mid-2000s.
teiferer 2 days ago [-]
Seems like they want to play in the big boys club (Google, Facebook, ... ) where this kind of treatment is standard.
bigfatkitten 21 hours ago [-]
The difference of course is that Facebook are experts at doing nothing at all about CSAM that you report to them.
ajjahs 2 days ago [-]
[dead]
gabucino 1 days ago [-]
[dead]
jstummbillig 1 days ago [-]
Imagine you are OpenAI, with hundreds of thousands of business customers and trillions of actions taken through your system every day, in violation of your policies and often local laws.
How much patience would you have for some entity maybe "accidentally" processing something that (apparently) looks like child porn against your policies?
Toynbeeidea 1 days ago [-]
I don't need to imagine. The postal service, telephone company, and internet service providers have all existed, and contnue to exist.
jstummbillig 1 days ago [-]
And you have insight into their level of tolerance for CSAM?
qingcharles 1 days ago [-]
This is understandable. I think the core problem is that most big tech/social media have no working appeals system which would allow a false-positive situation like this to be resolved in a sensible fashion.
antiloper 2 days ago [-]
I mean, you're holding it wrong. You uploaded CSAM to OpenAI which is presumably against their TOS.
harshreality 1 days ago [-]
So nobody can classify any user-submitted image content using cloud AI providers.
Is this really the world you want?
hackernud3s 1 days ago [-]
Are you sure you want to defend uploading CSAM.
abalashov 1 days ago [-]
Are you sure you want to shut down reasoned discussion of the pros vs. unintended consequences of a thing?
hackernud3s 1 days ago [-]
I don't think there's such thing as reasoned discussion on this topic. You're either against it, or you're Jeffrey Epstein.
24 hours ago [-]
mock-possum 22 hours ago [-]
How do you know it was uploaded?
2 days ago [-]
user43928 1 days ago [-]
Allegedly. Or the discrepancy in Cloudflare's and OpenAI's detection is caused by a false positive on OpenAI's side. Who knows?
gwd 1 days ago [-]
So I've been working on having Claude using GPT images to illustrate various fairy tales. At some point there was a Russian one of a girl left in the forest, but in the first cut she looked peaceful and happy. I told Claude I didn't think that was realistic, and it said that it had already asked GPT images to show the girl "scared", and it refused 12/12 times. It also refused to have Frost (who in the fairy tale ends up giving her a coat and a chest of gold) standing over her; he had to be at her level, an arm's length away. (Showing the girl as "brave" and then "nervous" had better results.)
Gave me quite a funny feeling, tripping innocently over the empty shape that they've had to carve out. They wouldn't have spent the effort to do that if it wasn't pretty bad.
Aeolun 1 days ago [-]
ChatGPT is only too happy to generate really messed up images of ambiguously aged characters though.
nolist_policy 2 days ago [-]
Relatedly, Openrouter now does CSAM scanning and retention on so called "Zero Data Retention" endpoints as well.
Every mainstream vendor with a US presence does this on their ZDR/MAM AI endpoints. This also includes AWS and Azure. It's explained directly in the AWS ZDR docs, and it falls under the "synchronous" scanning pipeline in Azure with full MAM.
At least with AWS they're clear that this is the only type of proactive scanning they do on their ZDR AI endpoints. With Azure it's actually unclear exactly how much more falls into the synchronous scanning pipeline, though I may be missing some docs.
The way this was explained to me is although there is no legal requirement to do this, vendors are concerned that AI processing constitutes "actual knowledge" of the content unlike a lot of traditional cloud-based workloads, which would expose them to legal liability. Seems like a weird argument in a ZDR context, but anyway, there's also a public optics element too I'm sure.
nwlieb 1 days ago [-]
I'm personally hopeful that as an industry we can move towards E2EE and attested secure enclave pipelines to remove the "actual knowledge" argument. NEAR AI and Tinfoil are the only publicly available providers in this area but I would be excited to hear about more advancements in this area because I personally think that these NCMEC integrated "ZDR" systems are major privacy violations and deceptive marketing.
AlanYx 1 days ago [-]
Azure does offer nVidia's secure enclave architecture via "Azure Confidential GPU", but it's limited to a single GPU and a little less than 90GB after overhead, so basically maxes out at Mistral-Large-2 with 4 bit quantization.
This whole thing kind of puts the industry in a bind. On one hand, the big labs do have an incentive to get this kind of thing rolled out so as to avoid open weight on-prem AI gaining traction, and to remove remaining roadblocks to corporate adoption in regulated industries. But rolling it out risks attracting even stronger regulatory responses for them as well (that's even setting aside likely national security/law enforcement pushback), as well as likely being strongly opposed by the AI safety people. There's no one way to square the circle that will please everyone. My guess is the labs will take a third route: continue to lobby against open weight models, thus killing the possibility of on-prem, and then drag their heels on E2EE/secure enclave models except perhaps for military customers.
nwlieb 1 days ago [-]
I'm hopeful that Tinfoil's enclave router, which does decrypt the payload but does so in a verifiably secure manner, can be used to scale to multi-gpu models (for example, Tinfoil does offer Kimi K3.1 Pro).
Because of economies of scale with batch inference as seen with DS4.1 Flash, this is the only avenue for true privacy from frontier models at scale/cost.
In terms of policy, I agree it's looking dire. There's unfortunately just more money/lobbying/mindshare on the 'safety at all costs' camp but I'm still advocating and educating where I can.
fc417fc802 1 days ago [-]
We will however need hardware elements that are actually secure. The current generation is trivial for the operator to compromise. https://tee.fail/
1 days ago [-]
itake 2 days ago [-]
How are these companies able to legally train these models? Are the employees citizens of countries that this material is illegal to download or possess?
qingcharles 1 days ago [-]
Most CSAM statutes have a carve-out exemption for things like this.
Phelinofist 1 days ago [-]
But they still have to obtain the images, look through the images and do tests with the images? Sounds like a totally fucked up thing to do. Reminds me of a fellow student that now works for a IT forensic company that does forensics for the police. He said that 98% of police requests involve checking storage of suspects for CP. He also has to look at the images/videos to write a report. He changed, no joke.
itake 1 days ago [-]
If the images are pre-labeled, you can look at the graph to see the training progress…
That being said, data science people might be curious as to which images it’s failing to detect…
intended 23 hours ago [-]
Doing safety work always changes you.
yh11 1 days ago [-]
i'm pretty sure certain organizations are given authorization
NooneAtAll5 1 days ago [-]
there never was a point in using rent-seeking middlemen anyway
abalashov 1 days ago [-]
Yeah, that's lovely. It's either ZDR or it's not, for Christ's sake.
"ZDR, except for _______" is not ZDR.
fc417fc802 1 days ago [-]
While I don't support such measures zero data retention does not mean zero data inspection so I don't think there's anything dishonest going on. Also it's generally to be assumed that any records believed to be linked to criminal activity are retained regardless of industry so again not an issue as far as the advertising goes.
abalashov 1 days ago [-]
I suppose you're right. It's hard to imagine them granting users unlimited capabilities.
Donald 2 days ago [-]
It is kind of wild reading these criminal affidavits around these National Center for Missing and Exploited Children integrations. The criminal uploads an image to their email, photo storage service, or gen AI service, it's detected as CSAM, automatically routes an alert to NCMEC, NCMEC generates a tip to the user's local law enforcement jurisdiction, and they're arrested within a short time period, like one or two weeks.
itake 2 days ago [-]
> A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal.
This is the problem. Unless an image is unambiguously a sexual image of a child it needs context, knowledge of intent, and common sense to classify correctly.
itake 1 days ago [-]
Companies declare that human moderators are augmented by AI, but humans make mistakes and even a photo lacks the context.
Maybe... its saying if you take the photo when the kid is in the bath tub or at the beach we ignore?
barbazoo 1 days ago [-]
> Family imagery: photos that are taken in private family settings where there may be instances of child nudity, such as a baby’s first bath.
I often take photos of my kids in the bath but they are neither babies nor is it their first so I guess that's CSAM then?!
teiferer 2 days ago [-]
There will only be very few generations of people who can see photos of them running around at the beach as small kids. That window closes faster than it opened just a few decades earlier.
abalashov 2 days ago [-]
I can see how this the average, unthinking member of the general population thinks this level of tech surveillance is a good idea, but those people haven't paused to consider the implications in literally any other context.
causal 1 days ago [-]
> reviewers confirmed 72% of the classifier’s high-confidence flags
They imply a 28% false-positive rate on their "high-confidence" flags. And that's an improvement? Yikes.
abalashov 1 days ago [-]
Yeah, and if you balk at this surreal conception of justice, you're just defending the "paedos", to many everyday people... good luck.
eutropia 2 days ago [-]
This is a positive development. People working for the NCMEC have had to deal with an insane increase in CSAM content (like upwards of 1,000% increase in 2025 alone) due to diffusion models dramatically multiplying the supply of CSAM.
When there's an active and urgent case to try and locate and rescue children in these situations, the volume of AI-generated content makes it that much harder to perform forensic investigations on background materials (often how they locate children being abused is by prosaic items like electrical outlets, household goods, etc. in the background of abuse photos and videos) because now they have to rule out whether an image is even real -- a task that often requires close attention and scrutiny to details of traumatizing and disturbing content (sometimes all the more traumatizing due to the 'creative freedom' of AI-generated material)
A model capable of shielding investigators from additional trauma is an excellent thing.
torginus 1 days ago [-]
CSAM stands for Child Sexual Abuse Material. Using diffusion models to create naked pics is by any reasonable definition not CSAM since no child is being abused. It's baffling to me that I have to state this, since it's tautological. The reason why child pornography is treated like a horrific crime by all of society, is because it harms actual children - this used to be the actual legal criteria for it.. Extending the reach of the crime to something that has none of that impact seems like a transparent power grab.
Just to counter any obvious moralizing, I am not saying using models like that is okay, but it's an entirely different kind of crime.
abalashov 1 days ago [-]
Well, the argument has been that it's a kind of vaguely cosmological or conceptual abuse of children, through the normalisation of sexualised depictions of children and so forth. In the production of such material, children suffer as a class.
The problem is, this viewpoint is infinitely flexible, infinitely elastic and can do a lot of gymnastics. I'm not sure it's compatible with the very concrete and procedural orientation of Anglo-American jurisprudence, particularly.
torginus 1 days ago [-]
I agree, there needs to be a concrete legal framework around this.
One closely related example was the punishment introduced for (legal-age) deepfake porn for which legislation already exists - this could be handled as a special (more severe) case of that more appropriately.
eqvinox 1 days ago [-]
> Using diffusion models to create naked pics is by any reasonable definition not CSAM since no child is being abused.
It might not be "CSAM", but it's quite illegal in Germany, Switzerland and Sweden, probably other countries too. If it looks like a child and is pornographic, it's illegal. Even comics and hand drawings fall under this.
lossyalgo 1 days ago [-]
Is it still allowed to use Grok to generate nearly naked images of children on X? Was X ever banned in any of these countries during that whole fiasco? I haven't followed the news but I remember it was a big thing for weeks. I presume it's still impossible to block people from doing that, as it's always been pretty easy to jailbreak LLMs.
abalashov 17 hours ago [-]
It's not, and public Grok has been greatly curtailed in regard to generating any sexualised imagery. However, I've read there's a thriving marketplace of rich prompting creativity in circumventing its guardrails, at least to some extent.
MrDrMcCoy 23 hours ago [-]
It should be illegal, but it should be a different illegal.
nonethewiser 1 days ago [-]
>The real question, whether indulging sexual desire for children is wrong even when no child is touched
Only if you think right/wrong sex is a function of consent and harm. Which is a very modern idea. And also cant condemn incest between consenting adults.
I see your perspective. It's a very coherent path to normalized ai generated child porn.
torginus 1 days ago [-]
I ask you to not reword my statements to make your arguments easier to win and argue for/against the points that actually were made.
The law already recognizes nuance. Calling for the death of someone is still a serious crime, but an entirely different ballpark to actual murder.
Saying that you need to persecute these people as a full-on child predators or not at all is not consistent with how basically all of existing laws work.
nonethewiser 1 days ago [-]
Im not rewording your statements to make it easier to argue with.
Which parts of my comment do you feel I mischaracterized you? Are you saying you think there may still be something wrong with child porn even when the subject doesn’t exist?
The law is already clear on computer generated child porn. Been explicitly illegal since 2003 if we are talking about the US. And shouldn’t it be?
belorn 1 days ago [-]
If we are talking about what should or should not be illegal, we need to first note that child rape is the only crime that is illegal to depict using computer graphics. It is legal to make a torture and murder simulator, with as realistic graphic and sound as is technical possible to make, including torturing and murdering children. Rape of adults are also legal to depict, but not of children. That is the exact point where the line is drawn. Same for video and images. The only exception is for books and music.
Should the line between illegal and legal be drawn exactly at that point, or should it be drawn somewhere else?
Manuel_D 19 hours ago [-]
In th US, at least, the Supreme Court decided in Ashcroft v. Free Speech Coalition that fictional depictions of minors engaged in sex acts is protected by the First Amendment. A few people have been prosecuted for it nonetheless, but in theory they would have prevailed if they appealed.
abalashov 18 hours ago [-]
This is not entirely accurate. A year after that decision, Congress came up with the PROTECT Act of 2003, which created 18 U.S.C. § 1466A[1]. The long and short of it is that this law aims to capture artificial sexually explicit depictions of children under "obscenity", and appeals under it have largely failed[2][3].
By and large, it is radioactively illegal to produce and transmit AI-generated child pornography at this time. Anderegg, a relatively narrow ruling in the 7th District, calls into question whether it is illegal to merely "possess" such content, but practically any enforcement of these laws turns on "production" (apparently a separate charge, which stood in Anderegg) and distribution (which also stood in Anderegg).
The "production" element is really key. Anyone who possesses such content either made it (production) or received it from someone else, which are both chargeable separately from possession. This effectively makes it illegal. It's like saying it's legal to have food in your mouth but not legal to put food in your mouth and not legal to "eat".
Separately, there's the matter of state laws. 48 states have _some_ kind of laws on this subject that supersede federal ones. It is overwhelmingly more likely that one will be arrested and charged based on those.
[2] US vs Whorley (4th Circuit, 2008), a case that involved Japanese anime cartoons depicting minors.
[3] US vs Arthur (5th Circuit, 2022), involving a web site of written stories that, in part or in whole, graphically depicted rape, murder and sexual abuse of children.
ogurechny 20 hours ago [-]
As long as the public takes the “but it can be used for child porn” bait with 100% success rate, no one is letting go of such a great tool to manipulate it. A technicality that would only affect “bad guys” is actually used to control everyone, wow, how new is that.
abalashov 17 hours ago [-]
It really, really doesn't help that Elon intentionally trolled them with a bot that can post sexualised deepfakes _publicly_, straight to X. That's just taking the piss, pouring gasoline on a fire.
himata4113 1 days ago [-]
People forget that less than 300 years ago people were r**ing children like it was normal. I'd argue that it survived up until half a millenia ago considering the fact that large groups (namely epstein) still exist and are surviving to this day.
rootusrootus 1 days ago [-]
Hell, wasn't CP basically legal* as recently as the 70s?
* in the sense that there was not many laws against it specifically; as I understand it the imagery may have been considered illegal under broader obscenity statutes
On the issue of Epstein, that probably would warrant more nuance. I would be willing to bet that many people still today draw a very significant distinction between the teenagers Epstein preferred and, for example, prepubescent children.
himata4113 1 days ago [-]
Statutory rape didn't exist until 1996, so yah you're right.
rootusrootus 1 days ago [-]
Heh, occasionally I get reminded that I grew up when many things we understand to be truth today were not in fact recognized as such then. Starting to feel old. I don't remember that the 80s were really that bad. Different time for sure.
pessimizer 1 days ago [-]
> I don't remember that the 80s were really that bad.
They weren't, people's brains have just rotted. They've started to call people who are attracted to 17 year olds pedophiles (rather than calling most people saying that they're not attracted to 17 year olds liars who you shouldn't be alone with), fighting for the right of 18 year olds to be prostitutes, and meanwhile you can get sued to sharing copyrighted videos of 18 year olds being paid $1200 to do triple-anal penetration by one of the people who was doing the penetration and also owns the copyright.
It's not a new moral framework, it's a bunch of slogans and a bunch of legal contradictions. And somehow, things become more legal and more moral when they're filmed or done in the name of commerce.
In the 80s, or to be more accurate pre-9/11, you were expected to be able to make a coherent argument about things that you believed. After 9/11, coherent arguments became a sign of privilege and possible foreign propaganda.
The kids born after 9/11 just didn't have a chance. We raised them to say the words, and told them if they interrogated the words that they would be expelled from school, fired, and socially shunned. We had eroded workers' rights (and worker's consciousnesses) so completely by then that firing anybody for anything was trivial. We had eroded free speech so much that any reaction to anything you had said or done was not only fair but actually good, and maybe obligatory.
Now, AI is helping (and being allowed to help) enforce this strategic, authoritarian confusion. I'm sure that will end well.
Facebook: it started off censoring nipples, and ended up censoring anti-wingsuit posts and wishes that people would die. Then it branched out into surveillance devices and nuclear reactors. Are you happy about how power develops in this new world, and how it starts with the centralized control of "truth"?
/rant
abalashov 1 days ago [-]
You might be my spirit animal. Definitely winning the Internet today, at least.
torginus 1 days ago [-]
> draw a very significant distinction between the teenagers Epstein preferred
That is an entirely different line of discussion, and please, please, please do not go there, as it's unlikely you'll find support for these opinions
pessimizer 1 days ago [-]
It's actually a great place to go. The idea that people who are attracted to 17 year olds are pedophiles is a corrosive idea. The idea that the reason pedophiles should be punished because they don't love the children is a corrosive idea. They're victories of mindless sloganeering over morality.
The reason that we have an age of consent is to protect people under the age of majority from being used as commodities by adults - to give them a little time before wiser, more powerful people start using them and trading them. It's the same reason why we have laws against child labor.
We don't have an age of consent because it's disgusting to be attracted to 17 year olds, and we don't have legal ages when children are allowed to work because they wouldn't be good workers before then. We do it because children should not be commodities until they've had enough training in this world to be able to defend and represent themselves to some extent. You should be working in a child's interest, they should not be working in yours.
Epstein exploited the girls that were over 18, who were the vast majority, too. They were marginal girls from broken homes who needed money. Making the odd cases when he was connected to 17 year olds the defining characteristic of what happened distorts and trivializes everything. It reminds me how #MeToo ultimately trivialized what Weinstein did by lumping it in with any sort of unwanted sexual advance. I've heard from more than one person that Weinstein went to jail for sexual harrassment. No, he was a serial rapist.
lossyalgo 1 days ago [-]
Let's not forget that Epstein did horrible things with many girls who were much, much younger as well. There are plenty of images and email correspondence that have come out indicating as much. I don't know how trustworthy NYPost is, but there are plenty of other articles that have gone through the emails.
> “You read through these files, and you read about 15-year-old girls, 14-year-old girls, 10-year-old girls,” Raskin, the top Democrat on the House Judiciary Committee, told reporters. “I saw a mention of a 9-year-old girl today. I mean, this is just preposterous and scandalous.”
As Judith Levine learned back in her day when she tried to publish "Harmful to Minors", this is an area where nuance goes to die. You're either for "paedophilia" (as defined by whoever's standing in front of you after having considered the matter for a few milliseconds), or against.
himata4113 1 days ago [-]
IMO there's far more harm being done by 'think of the children' laws than pedophelia has ever done. We've reached the point where people don't give a f*ck about children.
abalashov 1 days ago [-]
It's difficult to weigh those things, and I'm not an empirical researcher in this area. My gut sense, though, is that genuine paedophilia is quite rare, while the harms of thought policing and censorship are all too vivid and acutely present in the every day.
paulryanrogers 2 days ago [-]
> diffusion models dramatically multiplying the supply of CSAM.
Isn't CSAM only material based on abuse of real humans? How could computer models multiply CSAM by that definition?
schleck8 1 days ago [-]
Depends on the definition. It's become more frequent for predators to download images of real children from their parents' social media, then edit them with diffusion models to be nudes. This should still be illegal, and it is in the US.
> In People v. Gerber (2011, 196 Cal.App.4th 368), the California Court of Appeal overturned a conviction against a father who digitally pasted his 13-year-old daughter's face onto adult pornographic images. The court ruled that this did not violate California Penal Code §311.11, which criminalizes possession of child pornography involving a real minor "personally engaging" in sexual conduct. Since the images were digitally manipulated and did not depict the actual child involved in any sexual activity, the court held that they did not meet the legal definition of child pornography. The decision aligned with the U.S. Supreme Court's ruling in Ashcroft v. Free Speech Coalition (2002), reaffirming that purely virtual or fictional content—without the involvement of real minors—is protected by the First Amendment.
Most of the laws prohibiting deep fakes don't ban deep fakes on their own. They require specific other behavior, like using deep fakes to harass someone.
abalashov 18 hours ago [-]
This is true, although it seems that the common denominator is that transmission will suffice, regardless of whether there is an intent to harass, blackmail, defame, etc.
Since almost anything involving the Internet can be held to be transmission, and indeed, interstate commerce, it seems that doing anything other than literally generating it locally on your own hardware and viewing it privately within the confines of your own home can be a serious violation of the modal state law on this issue.
Where laws really do seem vague--to my untrained eye--is on what constitutes sufficiently "sexually explicit" imagery. The traditional child pornography laws are pretty blunt on this, and mean basically anything that could conceivably arouse, it seems.
Here in Georgia, for example, the controlling legislation on non-child deepfakes seems to be SB78 (transmission of deepfake imagery with intent to harass, or posting publicly in general), which says[1]: "transmitting or posting nude or sexually explicit photographs or videos for purposes of harassing the depicted person; to provide for an increased penalty for the electronic transmission or posting to a web page that is accessible to the general public..."
But I suspect a lot of cases under this would turn on what exactly "sexually explicit" means.
Legally, if the computer generated content looks too real, its still criminal.
fc417fc802 1 days ago [-]
Well it's against the ToS. Whether or not it's criminal (in the US, I don't know how europe handles it) depends on if it would qualify as "obscene" which means that looking realistic isn't actually relevant (it might seem bizarre but people have been sent to jail for cartoons).
Am4TIfIsER0ppos 1 days ago [-]
I assume Europe depends from country to country. In this one Simpsons porn is illegal. You heard that right. A bright yellow four fingered character is illegal to draw without clothes.
xp84 1 days ago [-]
Yup. This is such hysteria. Personally I’d argue that the existence of diffusion models “multiplying the supply of this material” means that the value of it goes way down, meaning it’s harder for the truly monstrous parties who have for decades harmed actual human beings to create that “content” now face collapsing value of what they make, which makes it far less lucrative to be in that business and should make it less attractive to get into it, and remove some incentive to make more and more of it.
This is no different than how something like basic poster design is not a real service you can sell anymore since ChatGPT can make a passable one for free.
If that kind of pervert can’t be simply fixed (or even exterminated) with a magic wand, it would be far better for our society if they’re looking at fake images than tempted to make their own by harming kids. But we as a society can’t accept that so we police this as aggressively as we did the kind that had an actual victim.
abalashov 1 days ago [-]
Some have argued the opposite, that "multiplying the supply of this material" further normalises sexual abuse of children, whereas before, it lurked in the shadows of the dark web and was treated like serious contraband, which imbued it with a certain gravity.
Not saying I agree with them, just that this is one of the main points trotted out in support of intrusively policing generation and editing models.
xp84 1 days ago [-]
(Recognizing that it's not your argument but responding to it in general)
It's still seen as extremely contraband. We're talking one of the most universal taboos of Western society. Heck, it's equally detested by the US Right and Left. How many things can we say that about?
I don't think anybody is going to suddenly think it's normal, cool, or acceptable to possess or view real or fake CSAM images or to abuse children, just because we aren't prosecuting people for using a diffusion model in a gross way.
abalashov 1 days ago [-]
I would agree with that, but plenty of campaigners beg plaintively to differ.
nozzlegear 1 days ago [-]
Not sure that multiplying the supply of content that pedophiles jerk it to is a great idea. Not only do you risk exposing people who (hopefully) never would've been exposed to it before, but research shows that increased access to CSAM can actually help normalize the abuse and support deviant behavior in online communities dedicated to it.
The reason why it's a specific problem is in my original comment: Signal to noise ratio.
The huge increase in volume of imagery is impeding operations to rescue children whom are actively being abused or held captive because every image, real or generated, takes up time and resources from the people trying to rescue the _actively abused children_
So your middlebrow refutal ("um ackshually no children are harmed when a diffusion model makes an image" ) of (one of) the many problems with additional CSAM volume is unconvincing and makes me think you've not actually considered the situation in any real detail.
fc417fc802 1 days ago [-]
According to your reasoning I'm left wondering, if the synthetic can't be readily distinguished from the genuine then what are you supposing would be the remaining motivation for criminal parties to produce and distribute such things? By producing and distributing they commit a very serious crime and for what personal benefit exactly?
abalashov 1 days ago [-]
I have to imagine, at least for some paedophiles, that the reality is an important of the arousal, in the same way that people who are into, say, cam models, for a rather specific reason -- it's a real, live person, at least in some sense.
I assume those very pathological folks are the real constituency for real-world CSAM.
fc417fc802 1 days ago [-]
Possibly, but if that's the logic then it seems to me that the people in question have little to no incentive to widely distribute (lots of risk with no upside) while those consuming won't be attracted to any large public venue where you can't readily differentiate the content. I'm also not sure how much interest such consumers would have in prerecorded content to begin with - consider that cam models provide an at least nominally interactive experience.
I'm entirely failing to understand the logic by which synthetic data would pose a widespread systemic problem for law enforcement.
abalashov 1 days ago [-]
I think the charitable version of the argument is that if the cops can't tell what's real and what's fake, they'll be spending a lot of time on fake stuff.
fc417fc802 1 days ago [-]
Right I get that, but it seems (AFAICT) to be entirely dependent on assuming that the criminals will be serving up and consuming a stream of mixed content. But that assumption appears to fall apart under even basic scrutiny since it seems that there are two disparate groups that will both be strongly motivated to avoid mixing content.
abalashov 1 days ago [-]
I've heard the argument that if they can't have one, they'll settle for the other, and that in general, any form of it is a gateway drug to ... something.
As a matter of common sense, it never cohered to me, but apparently people believe it, and there's just enough ambiguous research to lend some support to the talking points.
futune 23 hours ago [-]
I'd love for the buyers to not be able to tell what's real and what's fake, and having the fakes outcompete the real on account of economics.
LorenPechtel 1 days ago [-]
Which would easily be solved by having the models digitally sign their work.
But it's far more important to demonize pedophiles than reduce the harm. Same as the laws against drugs and prostitution. Meanwhile, we aren't willing to do anything about the vice that's showing itself quite problematic: gambling.
abalashov 1 days ago [-]
I highly doubt that can be forced upon the open weight models.
LorenPechtel 9 hours ago [-]
The signing would be in the code, not the model. It doesn't matter what model is used, all that you care about is whether the image came from an AI or a camera. The AI signs the image saying it came from an AI.
Since the signed ones would be legal and the unsigned illegal there would be no need to force the market.
fc417fc802 8 hours ago [-]
The AI will sign it? Surely you mean the inference engine? This is borderline technobabble and is laughably unrealistic.
Consider the effectiveness of a law stating that after using a model to generate an image failing to voluntarily watermark that image in some manner is a crime. And you expect that everyone would then magically follow such a law voluntarily?
By that logic if we simply outlaw murder it will no longer happen ... oh right, we already did that and things clearly don't work that way.
abalashov 8 hours ago [-]
Such regulations seem to work in China, where images generated by models carry quite a bit of metadata about where it came from. But it's China.
eutropia 1 days ago [-]
Offenders use GenAI models to wash "original" (harm-causing) CSAM in a way that evades some existing perceptual hash-based detection (PhotoDNA) and scrub forensic clues at a scale previously unavailable to most, if not all, offenders. Then those images are intermixed in existing distribution channels, contributing a massive torrent of confusing noise made from real harm.
Not to mention the rampant increase in "sextortation" and grooming of teenagers caused by the ability for offenders to deepfake images of minors they otherwise don't have access to. This too causes investigation problems, because the images are recognizably of real minors (that's the fucking point), but the children aren't currently missing. Still, when we are talking about kids aged 2-13 being actively abused or held captive to create real material, every minute counts. The time wasted (right now, not in some hypothetical future where your theoretical economic equilibrium of CSAM abundance is reached) is hurting real kids.
Furthermore, private darknet sites trading in CSAM operate a lot like exclusive torrent trackers where they require trading like for like / clout / reputation and providing new verified material. Monetary incentives are not what make those places tick.
If OAI's model can improve investigators' ability to wade through the ocean of depravity fast enough to rescue more kids, then it's a great thing.
intended 23 hours ago [-]
Community. Pedophile groups build communities to share and get access to content and to find people with similar inclinations as them.
Not everyone has the same ability to create content either, so requests can be made to other members in the community to nudity or create content according to some rubric.
simoncion 2 days ago [-]
> Isn't CSAM only material based on abuse of real humans?
If only.
The same "hard on crime" ratchet that happens to most every thing that's popular to punish and extremely unpopular to speak out against the tightening of said ratchet has happened here.
abalashov 1 days ago [-]
It is indeed a very dangerous ratchet. Who in their right mind would take an extremely unpopular position that is affront to the SafeTy of children?
And yet, we have no means of disputing the accuracy of these inscrutable classifiers, nor any reason to think that similar approaches won't be applied to other "objectionable" content in other contexts, as is currently the fashion in Europe in the UK with regard to anything that could be conceivably deemed to be inciting hate, animus or disorder.
Automated technological surveillance and censorship is really, really having a moment right now. I'm not holding my breath that the pendulum swings the other way, but I sure hope it does.
nonethewiser 1 days ago [-]
Huh. So you don’t think it’s morally wrong to generate child porn using AI?
Toynbeeidea 1 days ago [-]
I do not. I would much rather that those interested do exactly that rather than trade in the real thing.
Just don't put it in MY face.
estetlinus 1 days ago [-]
Are you not confusing your preferences with moral here? It is morally wrong because they depict children in sexual context. I mean these sickos come up with all kinds of things.
With your argument, why would it any child pornography be immoral? It’s just 0s and 1s in the computer, innit? They never took the photo. They just downloaded it from the dark web.
4star3star 1 days ago [-]
But wait, why is the bar different for this subject matter than for other depictions of immoral acts?
Creating a real video of a real person getting their head chopped off: someone got killed, so that's clearly a problem. Creating a fake video depicting the same decapitation in graphic realism: no one bats an eye - go eat popcorn while it happens on a big screen and laugh about it.
So, why is one violent, illegal depiction given a green light, but another is reasoned about completely differently? I think there may be an argument, there, which is why I'm asking, but it doesn't spring to mind. How would you explain the contradiction to a peaceful alien species who uses asexual reproduction (knows nothing about sex) and is shocked and horrified by the violence that passes as entertainment to us?
paulryanrogers 1 days ago [-]
> Creating a fake video depicting the same decapitation in graphic realism: no one bats an eye - go eat popcorn while it happens on a big screen and laugh about it.
I don't think the fake is legal if depicting the same act. One could be at least liable in a civil context.
Aeolun 1 days ago [-]
Have you seen some of these ‘saw’ movies?
paulryanrogers 19 hours ago [-]
Aren't the "Saw" movies entirely fictional? I.e. not depicting real crimes?
thereader12 1 days ago [-]
You edited your comment. Below is the version I replied to in case of further edits
> It is morally wrong because they depict children in sexual context. I mean these sickos come up with all kinds of things.
Can you tell me which ethical framework you are using to make your assertion? Right now it seems like an appeal to emotion, which is how we get things like the Patriot Act, KOSA, and all these id verification laws.
estetlinus 24 hours ago [-]
For context, I’m from Sweden so those frameworks say nothing. Maybe they’re the same as Chat Control etc?
What ethical frameworks(s) is ”you shouldn’t do that, that’s wrong”?
LorenPechtel 9 hours ago [-]
The vast majority of people find much of it repugnant (I won't say all because simple nudity is simple nudity, the fact that a pedophile gets off to it doesn't make it pornography) but where's the victim?
You want me to think an act should be illegal as opposed to simply repugnant, show me harm to someone who was not a consenting adult. Addictive things should have safeguards around them to keep people from being addicted through not realizing the danger, but they shouldn't be prohibited.
Viliam1234 1 days ago [-]
No need to signal your virtue, we are all just anonymous strangers on the internet.
Given the choice between someone doing actual child abuse and generating fake pictures on a computer, every time someone chooses the latter over the former, the world becomes a slightly better place.
You may argue that it would be even better to do neither, that's nice, it's called the Nirvana fallacy. That's already the law today, and people ignore it anyway.
estetlinus 1 days ago [-]
Maybe I am a victim of child abuse, maybe I am a pedophile — you have no idea, since we are strangers on the internet right? Don’t label it virtue signaling, please.
Those who choices is a false dichotomy. It is so much more nuanced than that, and your framing is wrong. I take it you don’t work with sexual abuse and sexual offenders?
freejazz 1 days ago [-]
What a bizarre strawman
abalashov 1 days ago [-]
I don't think OP thinks that. I hear him as pointing out that this question can't be considered in acontextual, ahistorical vacuum. It's a set of simultaneous equations, where the other equations and terms are of a more general form:
"So you don't think it's morally wrong to generate _______ using AI?"
"So you don't think it's morally wrong to _______ using AI?"
The matter can't be considered in isolation, and although anyone voicing a healthy fear of where this is going can be portrayed, through demagoguery, as "pro-CSAM" or "anti-children" at this juncture, the logical endpoint of this kind of surveillance is in far more unpalatable places.
As the good folks fighting--a losing battle, at present--for digital freedom and privacy know, guarding dearly held liberties means securing those liberties for criminals and antisocial pursuits, too. That's just the cost of freedom.
nonethewiser 1 days ago [-]
I don't really understand what you are saying.
To elaborate, I was reading it to mean that they hoped the generation would be allowed because it doesn't actually involve a human.
What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?
I guess calling it a cost recognizes there is something wrong with it at least.
abalashov 1 days ago [-]
> What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?
I don't have a great answer on whether it should be "allowed", but I see much wrong--as a matter of grave implications for freedom--with all the presently available and deployed techniques for "disallowing" it.
nonethewiser 1 days ago [-]
>I don't have a great answer on whether it should be "allowed"
Im not asking about if it should be allowed, I'm asking if you find anything morally wrong about making child porn via AI, or consuming it.
Why can't you just say "yeah it's morally wrong?" Why is that not obvious?
rootusrootus 1 days ago [-]
Why are you focused on whether one particular commenter has the correct answer to this question and not whether their concern about unintended consequences has merit?
abalashov 1 days ago [-]
I think that's just the nature of caustic moral litmus test issues, of which child sexual abuse is certainly the main one. Any qualms and you're "pro-child porn".
nonethewiser 1 days ago [-]
Thats not fair - I have quite specifically not done that.
Im acknowledging that hes concerned about unintended consequences. Now my question is if he finds anything morally wrong with ai generated child. A lack of an answer seems to indicate no but I’ll still entertain a more direct answer.
stale2002 1 days ago [-]
I think the point here is there you are basically dodging the point by bringing up stuff that's irrelevant to the point.
You could have just addressed his argument about side effects, but instead went on to the most nuclear toxic part of the discussion in order to get an own on him.
nonethewiser 1 days ago [-]
I asked if he thought it was morally wrong. Side effects are independent. The side effects could literally be the worst thing ever and it offers no insight on if he thinks AI generated child porn is wrong.
abalashov 17 hours ago [-]
They're not independent. Law is shaped by moral consensus, but moral consensus is also shaped by law. The obvious example is drugs: most people can agree drugs are, on the whole, bad for society, but there's little agreement on the extent to which users should be actively prosecuted.
pingou 17 hours ago [-]
Are side effects independent of morality? Isn't morality also often determined by the effects that each action has?
Do you think it is morally wrong to generate a violent image?
nonethewiser 1 days ago [-]
Because its not clear if he thinks the freedom in question includes ai generated child porn.
abalashov 1 days ago [-]
To be equally clear: I'm not sure if it does, either. I laid out some of the off-the-cuff chain of thought in my long comment.
nonethewiser 1 days ago [-]
>To be equally clear: I'm not sure if it does, either.
You misunderstand me. It's perfectly clear to me that AI generated child porn is immoral and should be illegal. So there is no "either."
That it's not clear -- whether AI generated child porn is immoral or not -- answers the question I posed to you by itself.
abalashov 21 hours ago [-]
It's not perfectly clear to me, no. I don't really know. I offered some thoughts on why it may or may not be. My ruminations were quite sincere, not an effort to avoid answering your question. I'm just not sure it can be answered as posed.
I personally am quite repulsed by child pornography, but am I prepared to universalise that for everyone when no child is being abused? I'm not sure. As I said in my long comment, I don't think there's a lot of precedent for that, outside of obscenity laws that apply to traditional visual art. We generally think of such laws to be on the wrong side of free speech today, though there are cross-cutting and contradictory trends in this area.
Beyond that, I am broadly supportive of electronic and digital freedom. There are very different kinds of things one can do with AI-generated erotic content, and there's a reason most prosecutions focus on distribution, revenge porn, blackmail, etc., rather than the mere fact of the generation _per se_. That aligns with my values. Maybe AI child porn is immoral, but telling people what visual concepts they can and cannot generate in the privacy of their own home for own gratification isn't moral either, to my way of thinking.
abalashov 1 days ago [-]
Offhand, I'm not sure what's special about it There are all kinds of horrors one can generate with AI: beheadings, murder fantasies, extreme sadism, material promotional of darkly extremist ideology, text and literature intended to mislead/manipulate/deceive/brainwash, depict or glorify self-harm and suicide, etc, for example, and one could argue that all of these further actual socially harmful impulses and behaviours, feed and nurture social clubs built around them, etc.
A number of European jurisdictions--and famously, the UK--do just that, and they aren't necessarily wrong. It is a tangible social good to keep the unthinkable cruelty of Nazi atrocities and the Holocaust alive in our collective and institutional memory. God forbid we ever forget these... actually, we seem to be well on our way, in that doom loop of history.
We haven't really had to confront, as a society, the ethics of photorealistic but utterly synthetic content, which can be generated quickly and at a very large scale, that contravenes our legal and social norms. There are clearly important nuances: purely synthetic content is not the same as deepfakes, which the courts have called "morphed" images.
I hear people who say it normalises sexual exploitation of children and makes us callous and indifferent to it. I hear people saying it's better than actual exploitation of real children, and that--reasoning from long-established free speech and privacy principles present elsewhere in our jurisprudence--the government shouldn't be telling people what they can enjoy in the privacy of their own home, provided that thing doesn't have a human victim, nor appears in any legally cognisable way to harm a class of victims (this is where much "hate speech" and "antisocial ideation" type laws dwell in the UK and EU).
I'm not entirely sure what to think. I don't have a great theory of mind for the kind of person who generates AI child porn. I'm sure many people think they do, but I can confidently say that I don't. Are they prone to real-life sex crimes against children? Is there a high degree of overlap with sundry groomers and predators? Is there a correlation between their interest in this and sexual abuse of their own family members? Their own children?
Or are they just curious techno-libertarians, maybe with a contrarian streak or a few unnerving kinks? What's a child in this context, anyway? Are we talking about toddlers or teenagers? Are they a child, or the kind of quasi-adult Judith Levine had in mind in Harmful to Minors (1994)? Is the perpetrator here more in the style of a 'predator', or just a 19 year-old horsing around? I don't really know what these folks are thinking or making, as I've never met anyone with such open proclivities. I'd guess, as with most things sexual, that many things are a lot more common than one might glean from the readily visible surface. I'd also guess that the truly depraved extreme stuff is at the margins. I think? I'm not a sex researcher.
Sex crime against children is a big dragnet, though: in my day, they put away Genarlow Wilson here in Georgia for 10 years:
Most people seem to agree today that this was an instance of justice run amok and massively cruel and unusual punishment. The Georgia Supreme Court did, too, if only by 4-3. However, I was in high school and college in those days, and back in the day, advocating for this teen earned one few friends, and a "paedo" label, even though we all had girlfriends and boyfriends when we were in high school and ... I guess we were just supposed to pretend Genarlow Wilson was some apex sex predator because he was black, not sure what the message was. It's hard for me to imagine that a similar spectrum of possibilities, and shifting norms, doesn't exist in the world of child porn, even though it's an area that rarely sees the light of any kind of nuance. I refuse to believe it's just this opaque monolith, just as I refuse to believe the socialist bloc was one univocal monolith. There have got to be things that common sense can agree are somehow worse than other things, even if "better" is not a word one would reach for.
Anyway, my intent isn't to muddy the waters with "troubled" distinctions, but rather to say that without that kind of theory of who we're talking about, some details to enrich a mental picture and a moral penumbra, I can't really say if I think it's wrong. I don't know who's doing what in your example, nor the why. I'm reduced to mostly talking in the narrowly procedural mood of Anglo-American jurisprudence, which deals with rights and harms and boundaries and definitions, not, by and large, a central idea of the good. That works in court, which deals with whether one is allowed to do something, but doesn't settle the moral question much.
My first instincts are quite liberty-leaning on this matter, and in a world that is substantially electronic, digital freedom may matter more than any other kind, and we clearly took it for granted in the free and open Internet of decades past. I can appreciate the arguments from the other side, but they all strike me as special cases of much larger principles at stake--it's just that nobody wants to impale themselves on the child porn question due to its incendiary character.
nonethewiser 1 days ago [-]
>We haven't really had to confront, as a society, the ethics of photorealistic but utterly synthetic content
Not true in terms of the law and philosophy. Congress passed a law already making child porn illegal for computer generated images of children.
In terms of social norms its also not novel. People have debated if pedophilia is morally wrong absent the harm to the child. Some people believe sex is right/wrong based on personal details like consent and harm and others factor in more natural order (such as roles - children, spouse etc). Maintaining integrity of roles makes it trivial to exclude incest and cases like this.
Manuel_D 18 hours ago [-]
> Congress passed a law already making child porn illegal for computer generated images of children
And the Supreme Court overturned it in Ashcroft vs. Free Speech Coalition
abalashov 1 days ago [-]
> Congress passed a law already making child porn illegal for computer generated images of children.
In a different time, place and era in technology, well before it was possible to create, at scale, lifelike and photorealistic images of almost anything, none of it incorporating real people. Back then, I think the idea was to criminalise well-done "Photoshop" jobs, or something like that.
The scale does matter. If computers, for example, become extensions of our imagination, so that we can wish a mental example into vivid, lifelike being with the touch of a button, and this becomes routine, then prohibiting a computer operation may tread, in many instances, close to prohibiting a category of expression, maybe even a category of thought (who knows how far the man-machine synthesis will go if Elon has his way...)
I don't have all the answers, I don't think Congress does, and I don't think you do.
nonethewiser 1 days ago [-]
I dont understand the point you are trying to make. The scale matters to what?
Im asking you, do you find anything morally wrong with AI generated child porn?
abalashov 1 days ago [-]
I tried to address that in my long comment, which I expanded in an effort to add nuance, but evidently did not get through.
The best I can do in trying to condense it is: "AI generated child porn" is hopelessly vague. What exactly do you mean?
If you're thinking, "what do you mean 'what do you mean?!'", please read my long comment.
nonethewiser 1 days ago [-]
Your very long comment contains no reasons why you think AI CSAM might be immoral and many reasons why it might not be. In addition you suggest many examples of CSAM are not actually CSAM, which is beside the point unless you think there is no such thing as child porn.
Surely you can grant that child pornography does exist, right? That's what I'm asking about. I'll take you at your word that you do not know, but judging off what you said I can't see a single reason why you think maybe it's even a little bad.
abalashov 20 hours ago [-]
Child pornography unquestionably exists, and I think it's bad, but the reasons why I think it's bad are mostly rooted in the traditional arguments about actual harm to children, rather than just that it's bad _ipso facto_ for the mere fact of its visual existence. I'm genuinely not sure how those apply to AI-generated porn, and don't have a strong opinion against it.
I don't know if that's the turn law is ultimately going to take, but its hand may be forced by the impracticality of the alternative. That's the only reason any of us have any privacy at all: it's mostly a technological feasibility question. As someone said elsewhere in this discussion, the state would absolutely read our thoughts if it were easy for them to do so. As a corollary to that, I'd say that as technology has made electronic surveillance easier, the state and private companies have both helped themselves gluttonously, and we've barely scratched the surface as laypeople.
The morality of digital freedom and privacy is heavily entwined with my moral consideration of absolutely any digital transgression. When the underlying mechanism is purely digital, the question itself is digital. This is not a question about child porn to me. I take for granted people will have varying views on it. Instead, it's a question of, "Is there anything that is so wrong that one should not be allowed to privately do it with their computer in a way that doesn't straightforwardly affect anyone else?" At first instance, I'd say no.
imtringued 18 hours ago [-]
His point is that a 20 year old woman's nudes might be perfectly legal, then you put them through an AI filter that produces exactly the same body shape and suddenly it's AI generated child porn.
As far as I understand the laws in e.g. Germany, this is not impossible.
The reason for that is you can prove the age of the real person to be 20, but you could not do that for the AI filter version because you have essentially erased the birth certificate.
This results in a problem: what is age in the context of fiction? The real world age of the character depends on the creation/publishing date, with all characters involved being the same real world age.
If you go by the fictional age, then you run into the issue that the author can manipulate the fictional age.
The only way to enact the spirit of the law is to ban the functional consequence. What effect does the content have and where does it derive the effect from? The conclusion is that age must be reverse engineered based on the body shape, proportions, height, etc, but this means a child is no longer an age.
By treating fiction and reality the same, we have opened up pandora's box. Banning CSAM no longer means banning literal child porn, it means banning abstract concepts that may be present anywhere even if no children are involved.
For me the solution is obvious. Everything is its own crime. Purely fictional CSAM (normalization) is a separate crime from how non-fictional CSAM (harm to child) is a crime. Deepfake CSAM (harassment) is a crime but it is neither purely fictional nor non-fictional, so it is another crime again. You see, everything can be a crime and not be the same crime.
The biggest problems come from treating things that are not the same as if they are the same.
Why treat someone running a drug lab, someone selling a drug [0] and someone consuming the drug as if they are all the same? Because it's about total surveillance and not really about CSAM. Any demarcation line that sets a boundary goes against the total surveillance goal.
[0] which can be synthesized via AI without your own lab
abalashov 17 hours ago [-]
I hadn't really thought about it that way, but that is a valid framing of some of the same problems I see. A world where anything can be trivially synthesised challenges the understanding of what is and isn't real, and what is and isn't child porn, too.
In the past, the answer was more straightforward: child porn is visual depiction of children being used for sexual gratification. Notwithstanding rare, labour-intensive and not especially photo-realistic artistic depictions, as well as some ambiguous cases about artistic license, by and large, child porn depicted things that actually happened.
In the generative AI world, that's not true. At the very least, it forces a reckoning with the question: what is the prohibition of child porn really about? Is it about preventing real-world harm to children, or is it about preventing manifestations of the concept of sexualisation of children, on the theory that its mere existence harms children in principle?
paulryanrogers 1 days ago [-]
I'd prefer it be illegal to generate porn that looks like children. I also understand that threshold can be difficult to define. What I'd heard most recently was courts said CSAM must involve real people or it's just free speech.
abalashov 1 days ago [-]
It's been reported and discussed that way, but the ruling was actually quite a bit more narrow and contingent.
This is a tendentious framing, but legally detailed:
> The most important point is also the easiest to lose in the headlines: the AI-CSAM ruling in the Anderegg case did not hold that AI-generated CSAM is generally protected by the First Amendment.
simoncion 17 hours ago [-]
Do you believe that it's morally wrong for a human to generate a text-only story that -among other things- depicts fairly-graphic consensual [0] sex between an early-bloomed fifteen year old and their graduated-college-at-nineteen science teacher? [1]
If you think that's morally wrong, what if we change the subjects to an enthusiastically-consenting fifteen year old and sixteen year old? What if the author of the story is themselves fifteen? What if they're writing fiction about them and their girlfriend based on their intense and very enthusiastic mutual attraction, with the full knowledge and consent of their girlfriend?
If you think everything in the second paragraph is morally wrong, do you think all written smut, erotica, and pornography is morally wrong, or just those that depict humans under the age of eighteen; regardless of whether or not the authors are themselves under the age of eighteen? If the latter, why do you believe that it's morally wrong for teenage lovers to write sexy stories about each other, and do you believe that it's morally wrong for those lovers to verbally tell stories about their sexual escapades to their teenage friends?
[0] ...given the context, this would almost certainly be considered dubcon...
[1] For clarity, I very very strongly believe that the acts depicted in the hypothetical work of fiction in this paragraph are not things that should be done by any humans. I'm asking very specifically about the creation of the work of fiction.
WheatMillington 1 days ago [-]
I cannot imagine being pro-child pornography.
abalashov 1 days ago [-]
Few people can. However, when it comes to digital freedom and privacy, you can't surgically cherry-pick what you're for and against. You need to consider larger principles, and operate in generalities.
WheatMillington 1 days ago [-]
You definitely can. It's technically and morally consistent to say "I support digital freedom, and I oppose the sexual abuse of children".
abalashov 1 days ago [-]
You can't say "I oppose the sexual abuse of children and support digital freedom, but nothing concrete follows from this whatsoever." What am I consenting to, in regard to my digital freedom, in voicing that sentiment?
1 days ago [-]
trollbridge 1 days ago [-]
Good. Then you wouldn’t be able to imagine being pro-not having a state sponsored surveillance agent on every phone and computer you have.
freejazz 1 days ago [-]
Yes, who will cry for the people making virtualized child pornography which is, of course, sourced from actual child pornography? WHO???
nonethewiser 1 days ago [-]
This is a real moral dilemma for people who think consent and harm are all that matters. Vitriolic disgust at the idea of generating child porn using AI, but no good way to explain why its wrong.
sebzim4500 1 days ago [-]
If victimless AI child porn is flooding the market and making it harder/impossible to monetise actual CP then isn't that a massive positive?
WheatMillington 1 days ago [-]
The motivation of people who make and consume this content is rarely monetary and almost always gratification. Do you really believe that a proliferation of child pornography will reduce the demand for child pornography, and reduce the abuse of children?
sebzim4500 23 hours ago [-]
Seems like pretty basic supply and demand, yes.
intended 22 hours ago [-]
That’s only if the demand is exclusively for images.
I’d guess this is The difference between a substitute and complimentary good.
More cigarette ads didn’t just appeal to people who like cigarette ads, but smokers as well.
sebzim4500 22 hours ago [-]
I'm assuming that AI will also be able to make videos that compete with actual CSAM if it can't already
intended 20 minutes ago [-]
Well, we never stop producing content. So even if production goes up, it doesn’t mean that demand gets saturated, which is your contention.
rootusrootus 1 days ago [-]
Perhaps it will make provably real CP that much more valuable?
echoangle 1 days ago [-]
Why would it be more valuable?
If the before is “only real” and the after is “real and generated”, why would real become more valuable? Supply is constant or higher, so the price should be constant or decrease.
abalashov 1 days ago [-]
Maybe in a strictly quantitative sense, that's true. But I could envision someone making the argument that, just as in a world of AI slop, authentic human writing is all the more precious, so verifiably real porn becomes a more desirable needle in an ever-larger haystack.
echoangle 18 hours ago [-]
I don't think human writing has become more valuable due to AI. The AI thing actually demonstrates how the cheaper alternative somewhat devalues the original thing, which would be a good thing for CSAM.
abalashov 17 hours ago [-]
I don't know if valuable is the right word. What is clear is that connoisseurs who value writing for authenticity, originality and the insight it offers into the mind of the creator seek it out, and that it is now harder to find, so they're happier to come upon it. Does that make it more valuable per unit of written word? I'm not sure.
intended 22 hours ago [-]
No, because it isn’t victimless.
It’s all types of porn, including deep fakes.
The effort to create content is negligible so everything has gone up.
kikdij 1 days ago [-]
They probably use "uncensored" local models, though. I doubt commercial models could ever be used for that.
itake 2 days ago [-]
My colleagues in Singapore trained pornography detection model.
Pornography is illegal to download or save in Singapore… I wonder how they were able to train a model without the data.
zelphirkalt 1 days ago [-]
Maybe they got it sent from outside the country, on fast disks, so that they neither need to download it, nor save it (again).
0xCMP 1 days ago [-]
As with CSAM, most laws care about possession AND transmission.
layer8 1 days ago [-]
Maybe they trained with live camera feeds. Data doesn’t need to be saved or downloaded in order to exist. :)
itake 1 days ago [-]
Streaming is also banned.
layer8 1 days ago [-]
Camera feeds aren’t automatically streaming.
npteljes 15 hours ago [-]
They could have used VDIs not hosted in Singapore, and use, but now view datasets that are confirmed to contain pornography by a third party.
Datasets then could have been acquired by torrenting or scraping a porn streaming site, while never actually viewing the training set to verify (nor the test data for the model).
Or they just did something illegal, or have gotten some exemption. :)
dmitrig01 2 days ago [-]
They must have amassed a large data set of things that are not pornography. Then trained the model on that. Anything that doesn’t match the model is pornography.
/s
agilob 1 days ago [-]
A reverse bloom filter
>A Bloom filter is a space-efficient, probabilistic data structure used to test whether an element is in a set. It can be “definitely not in the set” or “possibly in the set,” meaning false positives are possible but false negatives are not.
trekz 2 days ago [-]
Funniest thing I read all week lol
nonethewiser 2 days ago [-]
Not hotdog
seattle_spring 1 days ago [-]
Interestingly enough, the "not hot dog" app could do this. At least for about 50.26% of the world population.
AmazingTurtle 2 days ago [-]
reminds me of the case where a dude made pics of his kid to show the doctor and his google account got permanently banned for CSAM (google photos involved)
mrandish 1 days ago [-]
I guess it's sign of the times that upon seeing the headline "Strengthening the fight against online child sexual abuse", my first reflexive thought wasn't positive. Instead it was a jaded assumption this would be another perhaps well-intended effort likely to either go awry, further undermine the privacy and workflows of law-abiding users, or be largely ineffective - and quite possibly all three.
deltoidmaximus 16 hours ago [-]
I just assumed it was another cry to create a regulatory moat by banning any open models so everyone has to rent AI access from them. That's basically the only thing Altman ever has to say and he repeats it with a different angle every week.
RIMR 1 days ago [-]
This kind of thing is always an excuse for creating surveillance tools that scan your files and report you to the government. It starts with CSAM because they know you don't want to argue against CSAM detection, but you absolutely should.
I deal with a lot of user-generated content, and I have been using OpenAI as a tool to help find offending content. Now I see that if CSAM ever makes it into one of my systems, OpenAI is going to send the authorities after me if their model sees it too, so that's a dealbreaker for that use case, and it was a big one!
It's also wild that they are trying to detect and report AI-generated CSAM. This does absolutely nothing to help protect actual children.
nonethewiser 1 days ago [-]
>It's also wild that they are trying to detect and report AI-generated CSAM
Say more. You have no problem with AI generated child porn?
RIMR 8 hours ago [-]
Well, to be more precise, detecting real child porn is a matter of matching images to fingerprints of known CSAM. Essentially, the government has a massive database of these images for use as evidence as people are discovered and arrested, and they also contracted tech companies to make detection models. This massively helps in getting CSAM out of your systems automatically, and with minimal exposure.
But detecting purely AI-generated child porn requires something that actually scans your images and tries to determine what is in them. That is WAY more invasive than a fingerprint scan.
And, on top of that, targeting people for purely AI-generated child porn doesn't accomplish anything to protect children. It mostly just targets mentally ill people for incarceration. I don't think we handle this problem correctly as-is, but if we are going to be out there busting people for CP, I would prefer that we be focused on finding child abusers and bringing them to justice, not just rounding up people with problematic attractions. The only real exception is AI being used to "nudify" children. That's fucked up and targets real kids, but I don't know how you go about detecting this any less problematically than anything else.
At no point during this pursuit do I think it is appropriate for my AI agent to be investigating anything on my computer to report to the authorities, because there are plenty of use cases that involve incidental exposure to CSAM, and I don't need the cops coming to arrest me because I used Codex to clean up my social website's report queue, and it came across offending content.
And for the record, as someone who has had to clean social websites up, there is a huge difference between AI-generated CSAM, and real CSAM. The AI stuff is just gross, but the real stuff is traumatizing. It's a shame that AI isn't going to make this easier for me filter out, rather it just makes it riskier for me to use my computer at all.
estetlinus 1 days ago [-]
> if CSAN ever makes it into one of my systems
If that is a worry you carry around, I really hope you work harder to make it never be the case lol. FYI it’s not a normal thing to worry about.
trollbridge 1 days ago [-]
Any site that allows users to upload content has to deal with this.
estetlinus 24 hours ago [-]
Do you mean a grim cousin of the Time To Dick factor? I don’t think the solution is more lenient filters so that web app developers get an easier life tho.
trollbridge 18 hours ago [-]
Huh?
No, anyone that lets the public upload content will eventually have to deal with figuring out how to filter and remove objectionable content, ranging from copyrighted porn, copyrighted software, to CSAM.
AI companies should be providing workflows specifically for this, not shutting down accounts.
RIMR 8 hours ago [-]
Best take I've heard in this thread so far.
Being able to let an cold emotionless AI review all my users images, and just destroy the CSAM and forward it to NCMEC or whoever my AI provider partners with to make sure the data is captured and reported would be great. It could keep my website clear of illegal user content, keep child protection organizations and the authorities in-the-know, and prevent me from ever having to lay eyes on something that is going to give me nightmares.
RIMR 1 days ago [-]
If you run a website with user-generated content, you absolutely will encounter CSAM. It may not be "normal" in that it shocks the conscience, but it is absolutely normal in that every single website that accepts user-submitted content will, at some point, encounter CSAM.
It's been like that for the entire 20 years I have run these kinds of websites. It sucks, but it's just a sad reality of the world we live in.
rootusrootus 1 days ago [-]
Yeah I feel like many people live in a naive little bubble. I was running BBS's in the 80s/90s and bulletin boards in the 00s. It does not take very long to find out how dirty a significant fraction of the general population is. Porn is nearly instant if you don't moderate against it, and CSAM won't be far behind. It is tough not to become a misanthrope.
estetlinus 24 hours ago [-]
Poor you who have to moderate your web apps so that child porn doesn’t get uploaded.
imtringued 18 hours ago [-]
You forgot that "moderate" may mean upload all user input to a central server hosted by a government controlled or government hired organization.
1 days ago [-]
nazgulsenpai 1 days ago [-]
How else do you check user generated content other than checking user generated content?
callc 2 days ago [-]
Curious what people here think as digital technology makes it easier to spot indicators that someone may commit a crime.
I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.
How should society/law change in response? Is it purely a give and take between surveillance/right-to-not-be-watch and trying to prevent all bad things from happened before they happen?
nonethewiser 2 days ago [-]
>I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.
Privacy, unfortunately, is largely the result of it being impractical to violate it.
I see zero reason to believe that states wont read your thoughts if they can easily manage to.
bob1029 2 days ago [-]
There is a popular movie starring Tom Cruise that explores the consequences of this question.
VariousPrograms 1 days ago [-]
In Minority Report (spoilers), the precrime department is shut down because of a single error proving the system is fallible, thus making the plot preposterous in the age of AI.
mitthrowaway2 1 days ago [-]
Psycho Pass is perhaps a better depiction of what could happen to a society where the system is broadly accepted.
komodo99 1 days ago [-]
If we're wading into this medium, then Shimoneta ought be mentioned too, albeit in an absurdist manner filtered through a comedic lens.
deltoidmaximus 16 hours ago [-]
We literally have executed innocent people with our current system and not only is there little will to change this system but even stopping executions in response is still up for debate. It was actually preposterous when the film came out!
Dick's original story is actually much darker and more realistic in that the main character's motivation is as much about preserving the appearance of infallibility in the precrime system in the face of it's flaws as it is about proving his innocence.
Paracompact 1 days ago [-]
Haha. In the real world, it's closer to "one success justifies the program's existence" than "one failure closes it down."
abalashov 1 days ago [-]
Boy, is that ever true. Vide for example the gangsterism of the deportation campaigns under this administration. Somewhere, somehow, they managed to arrest some actually bad guys, and this apparently vindicates mass roundups, wanton thuggery and performative cruelty toward the other 99%.
plagiarist 1 days ago [-]
Philip K. Dick: In my book I invented the Precrime Division as a cautionary tale.
Tech Company: At long last, we have created the Precrime Division from classic sci-fi novelette The Minority Report.
nonethewiser 1 days ago [-]
The consent/harm sexual ethics framework has a problem here.
Most people think AI-generated child porn is wrong. But the consent/harm framework can't explain why AI generated child porn is wrong. Either you reach for indirect harms which can just be stipulated away (its not shared so police aren't confused, local model, etc.), or you have to argue in favor of AI generated child porn. In fact we have seen plenty of that here. And this framework has similar problems with necrophilia, bestiality, and adult incest.
A framework that allows for some natural order does not have this issue. If there is some natural order, and right/wrong are not just a function of preferences, you dont have to judge actions solely by effects. From that view, you can say sex has a proper place and purpose, and children have a proper role -- to be protected, raised, and not sexualized. Sexualizing a child violates that order whether or not a real child is ever involved. So the wrongness doesn't require any specific child to be harmed.
I think thats what most people already sense - that the wrongness was never only about harms to children.
haickernews 1 days ago [-]
Other people don't exist just to roleplay the version of reality you prefer. There's nothing wrong with avoiding some people and cultivating your own community and standards, but everyone shouldn't have to abide by your rules.
nonethewiser 15 hours ago [-]
You are talking about morality right? We already have shared laws that everyone has to follow, which I think is pretty fundamental and you'd likely agree with.
I would agree that not every moral/immoral thing needs to be enforced by law. I do think AI generated child porn should be illegal but I take the broader point that not every immoral thing should be illegal.
So to that extent I already agree with you that people should be able to gravitate towards their own communities. I will gravitate towards the community that doesn't tolerate ai generated child porn. Will you?
haickernews 11 hours ago [-]
No.
stickfigure 1 days ago [-]
How is this different from omni-moderation, which OpenAI already has?
Finally Sam Altman found something good to do in terms of my security, I'm relieved. It fits perfectly well in a world where Saudi Arabia lead the UN Commission on the Status of Women (CSW).
nrmitchi 1 days ago [-]
Frankly, and I don't mean this in a bad way about the problem, I do not think OpenAI actually gives a shit about the underlying problem. Especially as an OpenAI-hosted-service-only, this seems more like a data grabbing play. "Scan all your images through our CSAM detector" means "send all images to us, including the 99.99% that are not CSAM!"
Don't get me wrong: these are very necessary tools for the modern internet to function, and I'm sure _drastically_ reduce the review burden on humans, but the skeptic in me does not believe that OpenAI is being entirely altruistic here.
1 days ago [-]
andai 1 days ago [-]
>It is well established that combatting online child predators requires coordination across many organizations.
It requires all devices integrated into one single consciousness.
andai 19 hours ago [-]
Upon reflection, looks like it's probably going to be local LLMs -- one per device -- i.e. many fragmented "consciousnesses" functioning as informants.
sandboxdevelope 1 days ago [-]
Don't forget that OpenAI used CSAM to train it's models
bearjaws 2 days ago [-]
If only there was a huge data mine of child abusers sitting on some sort of email service they could go after. Maybe some sort of jeevacation inbox.
Could be a great start for "rogue AI"
Jokes aside, I'm glad they are giving tools to help fight this as currently your options are limited because every provider will simply return a content warning if you try to use an LLM as a moderator.
avazhi 1 days ago [-]
NCMEC is cancer.
Remember when they wanted Apple to scan everybody’s devices locally?
I’m not sure what the solution is to deal with pedophiles but I’m sure it starts with parents monitoring their kids better and keeping young children from being isolated from real world social groups. I’m also sure it doesn’t involve using classifiers on a person’s own device to detect ‘possible’ child porn, which is where this is headed. Who can forget the father who had photos of his sick baby on Google Drive for his paediatrician to see and got reported to the police.
If it really is all about protecting kids then maybe the NCMEC should be lobbying to pass laws banning kids from the internet. Think about how many harms you could reduce instantly (bye bye Instagram, Snapchat, etc).
kittikitti 1 days ago [-]
This is just public relations and no reasonable person would believe OpenAI will do anything to fight against online child sexual abuse. In fact, it's much more reasonable to assume they will continue to make the problem much worse.
formerlurker 1 days ago [-]
I agree with you. To me, it’s all lip service and nothing gets done until people get hurt.
Similar to Facebook and other platforms feeding self-harm content to vulnerable children and teenagers, and even connecting them to dealers. These companies know their platforms are not safe for kids. https://www.bloomberg.com/features/cant-look-away/
2 days ago [-]
rfgplk 1 days ago [-]
[flagged]
mx7zysuj4xew 1 days ago [-]
This is sarcasm right?
haickernews 2 days ago [-]
[flagged]
graemep 2 days ago [-]
Are you saying you are a paedo? I do not think you are going to get much sympathy for "discrimination".
haickernews 2 days ago [-]
[flagged]
graemep 2 days ago [-]
You are human, but you are not just talking about a sexual preference, you want to be allowed to act on it to at least the extent of being allowed child porn.
Whatever you say about "non-offending" (and possessing child porn IS an offence in most places). I would definitely not have wanted you anywhere near my kids.
haickernews 1 days ago [-]
As explained in another comment, this article includes examples of detecting drawings and AI generated images.
I believe AI is one of the only ways to engage with these desires in an ethical way.
Of course, you can always recommend abstinence, but unless you're rather conservative in your views, you probably don't think that's a good way to deal with teenage pregnancy, homosexuality, etc. So why would this be the exception?
rootusrootus 1 days ago [-]
> I would definitely not have wanted you anywhere near my kids.
Worth remembering, though, that the most likely offenders are family & friends. We spend a lot of time worrying about stranger danger because we cannot come to terms with just how common this predilection really is.
mcmcmc 2 days ago [-]
If you’re non-offending then why would you care about CSAM detection? Possessing CSAM is offending. You are not being discriminated against. That’s like saying murder laws are discriminatory against schizophrenics because it’s not their fault if they kill someone. It’s also incredibly damaging to the LGBTQ community to suggest that paedophilia is a sexual orientation when in reality it is a fetish/mental illness.
GalaxyNova 1 days ago [-]
While this is true it is difficult to argue that AI generated content or text hurts real children in the same way as CSAM.
mcmcmc 1 days ago [-]
Agree to disagree. If “non-offending” pedos can get jerkoff material it enables their fantasies and makes them more likely to prey on children in the real world. There’s also the prospect of non-consensual pornography, nudification apps etc - I’d argue it’s incredibly harmful to generate sexual imagery of a real child, even if they are not physically abused to create it
haickernews 1 days ago [-]
From the article:
>In SafetyKit’s evaluations involving drawings or animations that sexualize children, the classifier detected 90% of cases
>This is the first CSAM classifier with broad availability that has a specific signal on indicating if CSAM may be AI generated.
What children are being abused by drawings or AI generated images, stories, etc?
I understand the cases where it's problematic to make such content based on real children and share it, and I think it's probably a bad idea to allow photorealistic child porn to be publicly posted on the internet for the sake of not confusing law enforcement (in the same way posting misinformation is harmful), but if these images are not realistic, or if it's just text, or if they are realistic but they aren't being shared on the internet, who is being harmed?
nonethewiser 1 days ago [-]
Your perspective is entirely consistent with a consent/harm framework for sexual ethics. If it's all about consent and harm how is privately arranging and viewing 1s and 0s like this bad?
Do you also find these permissible, since there is no consent violation nor harm?
- Necrophilia
- Bestiality
- Adult incest
The problem with consent/harm framework is that you either have to start arguing some very indirect and abstract harm or you have to accept conclusions that everyone else thinks are just kind of fucked up to be consistent.
Once you introduce some form of natural law into your ethical framework it's trivial to say all these things are not permissible but I guess AI generated child porn and necrophilia is preferable to natural law to many.
haickernews 1 days ago [-]
Necrophilia harms the loved ones of the deceased by desecrating a dead body. Though the dead are non-entities and can't suffer, I still think it would be disrespectful to have sex with a dead body if it was against their wishes in life.
Bestiality is obviously harmful to animals. Best case dubious if you assume animals could consent and would be willing, but seems like there's little evidence for that.
Adult incest is only harmful if it leads to pregnancy and birth defects.
Finding something gross doesn't mean it's immoral.
I can understand the arguments others have about AI generated child porn creating more demand, but I think we're rapidly approaching a world where human beings can engage in any kind of (what would otherwise be harmful) behavior in VR, assuming we don't all die from misaligned AI. I would prefer to live in a world where I can fulfill my desires in a harmless way, even if they're abhorrent to others, rather than a world of value lock-in based on antiquated views of harm.
nonethewiser 15 hours ago [-]
>Necrophilia harms the loved ones of the deceased by desecrating a dead body.
In the scenario where the dead body doesn't have any loved ones whats wrong with necrophilia?
>Adult incest is only harmful if it leads to pregnancy and birth defects
And when it doesn't lead to pregnancy and birth defects? Are you OK with incest then?
haickernews 11 hours ago [-]
Personally, it seems polite to allow someone to have autonomy over their dead body as a last wish. Necrophilia would be disrespectful in the same way an organ donor would likely not want their cadaver used for testing weapons. If you could "3D print" cadavers with others likenesses, performed sexual acts on it, and did not let information leak to the outside world, I think this would be fine.
Regarding incest, yes. I don't see the harm.
Computer0 2 days ago [-]
For anyone else who doesn’t know what that is: NOMAP stands for "non-offending minor-attracted person."
nonethewiser 2 days ago [-]
Aka pedophile
I understand the distinction between offending and non offending.
But pedophile is inclusive of both.
rootusrootus 1 days ago [-]
I wonder, if you asked 1000 random people, what fraction of them would recognize that distinction?
We asked for further information so we could report the uploader to the NCMEC and remove the content from our systems. They refused to give us any information about this incident, denied our appeal, and have so far, illegally refused our privacy requests for data about our account.
OpenAI does not care about fighting child sexual abuse, this is completely performative.
EDIT: These image uploads are up to 20+ years old, and are being served through Cloudflare with their CSAM scanning product enabled. We've received no notification of any problems from Cloudflare. Something on OpenAI's system thinks that an image we sent to them for classification is CSAM, and they aren't willing to elaborate or communicate about it at all. For reference, our website is https://ytmnd.com
Another complicating factor is that it really blows to have humans look at stuff like that, so everyone would rather use automated classifiers; and if you assume good faith and share too much info with the affected parties, this will be exploited by the bad guys to gain insights into how the system works and how to circumvent it.
I don't think there's an elegant solution to this, other than "don't allow tech companies to get so big that they can afford not to care about individual customers".
But as a matter of self-preservation, their policy is basically "don't ever bring CSAM to any of our platforms, period". It's the same as Google, Apple, Meta, etc. It doesn't matter if you're a researcher, an investigator, if it's an accident. They will ban you because they don't want to be in the business of adjudicating this stuff and because your account doesn't matter to them.
It's completely orthogonal to caring or not caring about the issue.
I agree, almost. The risk isn't letting CSAM through though, the risk is "being seen as not doing enough". And being a hyper-anal mechanical pharisee is good for being seen as doing something.
On the contrary, that's an incredibly elegant solution.
I don’t think this is something that can be solved by more surveillance though.
Most abuse (i think something like 80%? Not sure but it was a big number) happens within the family and between minors (minors abusing minors)
Changing how easy or hard it is to distribute or generate CSAM will help exactly nothing here even if that’s the goal.
AI unfortunately currently can't be e2e encrypted at scale, so the the big players are forced to use surveillance to protect their legal butts from being sued.
Imagine if the government (USPS) decided they need to open and read every single letter or package you send or receive, for /safety/.
It can actually, there are even inference providers that offer this. It isn't strictly E2EE but running a published image on attested hardware is the next best thing. The customer can inspect the image and the hardware manufacturer attests that this is the image that the inference provider is actually running.
That said the latest generation of attested hardware is quite vulnerable (see https://tee.fail/) so you're still taking the provider at their word that they haven't intentionally compromised their own machines.
Of interest for those who might not know, the USPS does this for _metadata_ today.
The US Postal Service photographs every single article of mail it processes in the US through a program called Mail Isolation Control and Tracking[0]. It was created post-2001 anthrax scare, and is widely shared across LEOs.
But unlike most LEO intercept systems, you can actually /benefit/ from this one and get your own lens into their data feed.[1]
Modern problems, modern solutions, etc.
[0]: https://en.wikipedia.org/wiki/Mail_Isolation_Control_and_Tra...
[1]: https://www.usps.com/manage/informed-delivery.htm
Basically, keeping children from being miserably. Poverty isn't the only source of the exploitation of children but it's huge one and how little any anti-CSM crusaders care about this reveals a lot imo about their lack of sincerity.
But in our society, even in a much better society than this, bad things are going to happen at scale. At a certain point, you have abandon the "we'll do anything to stop this terrible thing, no matter how much what you do hurts people otherwise, no matter how ineffective what you do is, you must do your thing since it can be done".
> What is appropriate to what age?
Pretty much everything. You can talk to children about sex without issues.
> Who should do it?
A person the child can trust, can be a teacher or parent for example
> How much should it be driven by children's questions and how much regardless?
If they have questions answer them.
> Should all children by taught the same things at the same age?
Yea I don’t see why not
> Should it be purely biological or do you teach them values at the same time?
Both
If you look at sibling comments they disagree with you. One assumes a syllabus taught by teachers and says values should not be taught. If you do teach values whose values? There is nothing people disagree about more than sex.
You say "a teacher or a parent" for example. If its a politically imposed system it has to be the former. I think it is better for it to be the latter.
Children are very different so they should not be taught the same thing at the same age. Its a general flaw of classroom education but its even more of a problem with something where maturity matters. I disagree with everything being appropriate at any age - you notice GP said "age appropriate".
We have had as many different views in this thread as people have commented. I think that proves my point.
It is in a democracy
> If you don't teach your kid about body safety and autonomy
I agree, but even people who agree will disagree vehemently about the details when you try and turn it into policy. Lots of people will want to add stuff. Lots of people will want to remove particular things. lots of people will want things taught differently or a a different age.
> It is in a democracy
No, no man. We don't say, "look, most of us think that enslaving Group X is morally bad, but hey some people will disagree, so we're regrettably prevented from forbidding it." On the other hand, we do say, in several states, "listen, you may be a doctor who thinks abortion is a human right, but we absolutely require under the threat of delicensure and imprisonment that you try and talk women out of them."
People will be as nuts as you let them. There's no reason to tolerate this extreme stuff, especially to the detriment of us all. What's that meme, men will do anything to avoid going to therapy, the internet will require age verification DNA scans to avoid telling their kids about sex.
In veil of ignorance Rawlsian style you teach them enough where if you were in their position you wouldn't feel slighted by the adults setting you up for the world at large.
Keep values out of any mass education.
But this agenda is also ultimately about the power of those pushing it rather than just its goals, of course.
I didn't tweet, I wrote them an articulated email exposing the issue.
I think virtually every person and company does care. In fact I think its exceedingly obvious.
Even if OpenAI, Facebook, etc dont care be cause they are evil, they do care because pf liability.
None of this even excludes the possibility that CSAM controls are co-opted for power sometimes. Which is why it’s absurd to suggest these companies don’t care - you can concede that point and still argue its being co opted.
To actually care about something is not the same as doing arbitrary actions due to financial reasons. That's caring about money, and nothing else.
What do you think? Do you think Facebook, OpenAI, etc. are fine with child porn on their platforms save for liability? A liability which of course exists because virtually everyone agrees that it's a real problem and not just an excuse to grab power.
What makes you think same thing won't happen again ?
The fact that they did not previously have proper procedures in place for something like this is not proof that they don’t care. They are one of the fastest growing companies of all time, it’s hard to get disparate parts of an organization aligned on something. Their organization has more than doubled in size in the last nine months.
That doesn’t really follow from your experience.
OpenAI could care about child sexual abuse and not respond to your appeal, even wrongfully.
Frankly I am not interested in retaining an OpenAI account anymore so the only reason we appealed was to try to get more information, not to restore the account.
At the very least, they could provide us with an NCMEC case number so we can proactively work with the NCMEC to track the content down, but it appears their main motivation is not "strengthening the fight against child sexual abuse" but rather "reducing OpenAI's liability in the fight against child sexual abuse".
If you doubt this, consider that the "report" button on any website does pretty much exactly the same thing at the end of the day.
Are there any resources or tools you'd recommend here for me to setup? It's already going through cloudflare, and I'm actually in the process of shifting my transform service from Supabase to a hosted container there, so I've got my images routing through there anyways (but storage on Supabase still instead of R2 for now).
I've so far managed to resist using OpenAI and similar for anything, except having played around with duck.ai a little bit. I'd only want to use local models, preferably models that I trained myself. "Open weight" seems like perverse terminology, like "open binary" for closed source software.
Duck roll…
So basically, table stakes for running a user-submitted-content site will be to pay for your own GPU node/cluster and manage your own inference stack (running a worse model than you could use in the cloud). Otherwise you're at the mercy of opaque and kafkaesque (by necessity) service termination without warning for CSAM (real or hallucinated), and subsequent refusal to communicate about anything related to it.
[1] either by queuing all posts for classification before showing them publically, or just best-effort after-the-fact detection of spam or abuse, and upon detection they're moved to a spam queue.
Or just use one of the many third party inference providers that markets an image classification endpoint intended for user generated content.
The fact that people have autonomous agents making these kinds of decisions is still mind-boggling to me. Maybe I'm stuck in the past. But could this agent have been racking up things of dollars in charges because of this decision GP didn't even know about?
stay away, use deepseek and stuff
I used to view shit there a lot sometime in the mid-2000s.
How much patience would you have for some entity maybe "accidentally" processing something that (apparently) looks like child porn against your policies?
Is this really the world you want?
Gave me quite a funny feeling, tripping innocently over the empty shape that they've had to carve out. They wouldn't have spent the effort to do that if it wasn't pretty bad.
https://openrouter.ai/blog/announcements/csam-screening/
At least with AWS they're clear that this is the only type of proactive scanning they do on their ZDR AI endpoints. With Azure it's actually unclear exactly how much more falls into the synchronous scanning pipeline, though I may be missing some docs.
The way this was explained to me is although there is no legal requirement to do this, vendors are concerned that AI processing constitutes "actual knowledge" of the content unlike a lot of traditional cloud-based workloads, which would expose them to legal liability. Seems like a weird argument in a ZDR context, but anyway, there's also a public optics element too I'm sure.
This whole thing kind of puts the industry in a bind. On one hand, the big labs do have an incentive to get this kind of thing rolled out so as to avoid open weight on-prem AI gaining traction, and to remove remaining roadblocks to corporate adoption in regulated industries. But rolling it out risks attracting even stronger regulatory responses for them as well (that's even setting aside likely national security/law enforcement pushback), as well as likely being strongly opposed by the AI safety people. There's no one way to square the circle that will please everyone. My guess is the labs will take a third route: continue to lobby against open weight models, thus killing the possibility of on-prem, and then drag their heels on E2EE/secure enclave models except perhaps for military customers.
Because of economies of scale with batch inference as seen with DS4.1 Flash, this is the only avenue for true privacy from frontier models at scale/cost.
In terms of policy, I agree it's looking dire. There's unfortunately just more money/lobbying/mindshare on the 'safety at all costs' camp but I'm still advocating and educating where I can.
That being said, data science people might be curious as to which images it’s failing to detect…
"ZDR, except for _______" is not ZDR.
https://www.nytimes.com/2022/08/21/technology/google-surveil...
https://go.gale.com/ps/anonymous?id=GALE%7CA653985390&sid=go...
Maybe... its saying if you take the photo when the kid is in the bath tub or at the beach we ignore?
I often take photos of my kids in the bath but they are neither babies nor is it their first so I guess that's CSAM then?!
They imply a 28% false-positive rate on their "high-confidence" flags. And that's an improvement? Yikes.
When there's an active and urgent case to try and locate and rescue children in these situations, the volume of AI-generated content makes it that much harder to perform forensic investigations on background materials (often how they locate children being abused is by prosaic items like electrical outlets, household goods, etc. in the background of abuse photos and videos) because now they have to rule out whether an image is even real -- a task that often requires close attention and scrutiny to details of traumatizing and disturbing content (sometimes all the more traumatizing due to the 'creative freedom' of AI-generated material)
A model capable of shielding investigators from additional trauma is an excellent thing.
Just to counter any obvious moralizing, I am not saying using models like that is okay, but it's an entirely different kind of crime.
The problem is, this viewpoint is infinitely flexible, infinitely elastic and can do a lot of gymnastics. I'm not sure it's compatible with the very concrete and procedural orientation of Anglo-American jurisprudence, particularly.
One closely related example was the punishment introduced for (legal-age) deepfake porn for which legislation already exists - this could be handled as a special (more severe) case of that more appropriately.
It might not be "CSAM", but it's quite illegal in Germany, Switzerland and Sweden, probably other countries too. If it looks like a child and is pornographic, it's illegal. Even comics and hand drawings fall under this.
Only if you think right/wrong sex is a function of consent and harm. Which is a very modern idea. And also cant condemn incest between consenting adults.
I see your perspective. It's a very coherent path to normalized ai generated child porn.
The law already recognizes nuance. Calling for the death of someone is still a serious crime, but an entirely different ballpark to actual murder.
Saying that you need to persecute these people as a full-on child predators or not at all is not consistent with how basically all of existing laws work.
Which parts of my comment do you feel I mischaracterized you? Are you saying you think there may still be something wrong with child porn even when the subject doesn’t exist?
The law is already clear on computer generated child porn. Been explicitly illegal since 2003 if we are talking about the US. And shouldn’t it be?
Should the line between illegal and legal be drawn exactly at that point, or should it be drawn somewhere else?
By and large, it is radioactively illegal to produce and transmit AI-generated child pornography at this time. Anderegg, a relatively narrow ruling in the 7th District, calls into question whether it is illegal to merely "possess" such content, but practically any enforcement of these laws turns on "production" (apparently a separate charge, which stood in Anderegg) and distribution (which also stood in Anderegg).
The "production" element is really key. Anyone who possesses such content either made it (production) or received it from someone else, which are both chargeable separately from possession. This effectively makes it illegal. It's like saying it's legal to have food in your mouth but not legal to put food in your mouth and not legal to "eat".
Separately, there's the matter of state laws. 48 states have _some_ kind of laws on this subject that supersede federal ones. It is overwhelmingly more likely that one will be arrested and charged based on those.
[1] https://www.govinfo.gov/content/pkg/USCODE-2024-title18/pdf/...
[2] US vs Whorley (4th Circuit, 2008), a case that involved Japanese anime cartoons depicting minors.
[3] US vs Arthur (5th Circuit, 2022), involving a web site of written stories that, in part or in whole, graphically depicted rape, murder and sexual abuse of children.
* in the sense that there was not many laws against it specifically; as I understand it the imagery may have been considered illegal under broader obscenity statutes
On the issue of Epstein, that probably would warrant more nuance. I would be willing to bet that many people still today draw a very significant distinction between the teenagers Epstein preferred and, for example, prepubescent children.
They weren't, people's brains have just rotted. They've started to call people who are attracted to 17 year olds pedophiles (rather than calling most people saying that they're not attracted to 17 year olds liars who you shouldn't be alone with), fighting for the right of 18 year olds to be prostitutes, and meanwhile you can get sued to sharing copyrighted videos of 18 year olds being paid $1200 to do triple-anal penetration by one of the people who was doing the penetration and also owns the copyright.
It's not a new moral framework, it's a bunch of slogans and a bunch of legal contradictions. And somehow, things become more legal and more moral when they're filmed or done in the name of commerce.
In the 80s, or to be more accurate pre-9/11, you were expected to be able to make a coherent argument about things that you believed. After 9/11, coherent arguments became a sign of privilege and possible foreign propaganda.
The kids born after 9/11 just didn't have a chance. We raised them to say the words, and told them if they interrogated the words that they would be expelled from school, fired, and socially shunned. We had eroded workers' rights (and worker's consciousnesses) so completely by then that firing anybody for anything was trivial. We had eroded free speech so much that any reaction to anything you had said or done was not only fair but actually good, and maybe obligatory.
Now, AI is helping (and being allowed to help) enforce this strategic, authoritarian confusion. I'm sure that will end well.
Facebook: it started off censoring nipples, and ended up censoring anti-wingsuit posts and wishes that people would die. Then it branched out into surveillance devices and nuclear reactors. Are you happy about how power develops in this new world, and how it starts with the centralized control of "truth"?
/rant
That is an entirely different line of discussion, and please, please, please do not go there, as it's unlikely you'll find support for these opinions
The reason that we have an age of consent is to protect people under the age of majority from being used as commodities by adults - to give them a little time before wiser, more powerful people start using them and trading them. It's the same reason why we have laws against child labor.
We don't have an age of consent because it's disgusting to be attracted to 17 year olds, and we don't have legal ages when children are allowed to work because they wouldn't be good workers before then. We do it because children should not be commodities until they've had enough training in this world to be able to defend and represent themselves to some extent. You should be working in a child's interest, they should not be working in yours.
Epstein exploited the girls that were over 18, who were the vast majority, too. They were marginal girls from broken homes who needed money. Making the odd cases when he was connected to 17 year olds the defining characteristic of what happened distorts and trivializes everything. It reminds me how #MeToo ultimately trivialized what Weinstein did by lumping it in with any sort of unwanted sexual advance. I've heard from more than one person that Weinstein went to jail for sexual harrassment. No, he was a serial rapist.
> “You read through these files, and you read about 15-year-old girls, 14-year-old girls, 10-year-old girls,” Raskin, the top Democrat on the House Judiciary Committee, told reporters. “I saw a mention of a 9-year-old girl today. I mean, this is just preposterous and scandalous.”
https://nypost.com/2026/02/10/us-news/unredacted-jeffrey-eps...
Isn't CSAM only material based on abuse of real humans? How could computer models multiply CSAM by that definition?
> In People v. Gerber (2011, 196 Cal.App.4th 368), the California Court of Appeal overturned a conviction against a father who digitally pasted his 13-year-old daughter's face onto adult pornographic images. The court ruled that this did not violate California Penal Code §311.11, which criminalizes possession of child pornography involving a real minor "personally engaging" in sexual conduct. Since the images were digitally manipulated and did not depict the actual child involved in any sexual activity, the court held that they did not meet the legal definition of child pornography. The decision aligned with the U.S. Supreme Court's ruling in Ashcroft v. Free Speech Coalition (2002), reaffirming that purely virtual or fictional content—without the involvement of real minors—is protected by the First Amendment.
Most of the laws prohibiting deep fakes don't ban deep fakes on their own. They require specific other behavior, like using deep fakes to harass someone.
Since almost anything involving the Internet can be held to be transmission, and indeed, interstate commerce, it seems that doing anything other than literally generating it locally on your own hardware and viewing it privately within the confines of your own home can be a serious violation of the modal state law on this issue.
Where laws really do seem vague--to my untrained eye--is on what constitutes sufficiently "sexually explicit" imagery. The traditional child pornography laws are pretty blunt on this, and mean basically anything that could conceivably arouse, it seems.
Here in Georgia, for example, the controlling legislation on non-child deepfakes seems to be SB78 (transmission of deepfake imagery with intent to harass, or posting publicly in general), which says[1]: "transmitting or posting nude or sexually explicit photographs or videos for purposes of harassing the depicted person; to provide for an increased penalty for the electronic transmission or posting to a web page that is accessible to the general public..."
But I suspect a lot of cases under this would turn on what exactly "sexually explicit" means.
[1] https://www.legis.ga.gov/legislation/59239
https://www.bloomberg.com/features/2026-grok-sexualized-chil...
https://www.ibtimes.co.uk/xai-lawsuit-grok-ai-generated-chil...
https://globalnews.ca/news/12077556/sherwood-park-artificial...
This is no different than how something like basic poster design is not a real service you can sell anymore since ChatGPT can make a passable one for free.
If that kind of pervert can’t be simply fixed (or even exterminated) with a magic wand, it would be far better for our society if they’re looking at fake images than tempted to make their own by harming kids. But we as a society can’t accept that so we police this as aggressively as we did the kind that had an actual victim.
Not saying I agree with them, just that this is one of the main points trotted out in support of intrusively policing generation and editing models.
It's still seen as extremely contraband. We're talking one of the most universal taboos of Western society. Heck, it's equally detested by the US Right and Left. How many things can we say that about?
I don't think anybody is going to suddenly think it's normal, cool, or acceptable to possess or view real or fake CSAM images or to abuse children, just because we aren't prosecuting people for using a diffusion model in a gross way.
https://link.springer.com/article/10.1007/s11896-023-09611-4
The huge increase in volume of imagery is impeding operations to rescue children whom are actively being abused or held captive because every image, real or generated, takes up time and resources from the people trying to rescue the _actively abused children_
So your middlebrow refutal ("um ackshually no children are harmed when a diffusion model makes an image" ) of (one of) the many problems with additional CSAM volume is unconvincing and makes me think you've not actually considered the situation in any real detail.
I assume those very pathological folks are the real constituency for real-world CSAM.
I'm entirely failing to understand the logic by which synthetic data would pose a widespread systemic problem for law enforcement.
As a matter of common sense, it never cohered to me, but apparently people believe it, and there's just enough ambiguous research to lend some support to the talking points.
But it's far more important to demonize pedophiles than reduce the harm. Same as the laws against drugs and prostitution. Meanwhile, we aren't willing to do anything about the vice that's showing itself quite problematic: gambling.
Since the signed ones would be legal and the unsigned illegal there would be no need to force the market.
Consider the effectiveness of a law stating that after using a model to generate an image failing to voluntarily watermark that image in some manner is a crime. And you expect that everyone would then magically follow such a law voluntarily?
By that logic if we simply outlaw murder it will no longer happen ... oh right, we already did that and things clearly don't work that way.
Not to mention the rampant increase in "sextortation" and grooming of teenagers caused by the ability for offenders to deepfake images of minors they otherwise don't have access to. This too causes investigation problems, because the images are recognizably of real minors (that's the fucking point), but the children aren't currently missing. Still, when we are talking about kids aged 2-13 being actively abused or held captive to create real material, every minute counts. The time wasted (right now, not in some hypothetical future where your theoretical economic equilibrium of CSAM abundance is reached) is hurting real kids.
Furthermore, private darknet sites trading in CSAM operate a lot like exclusive torrent trackers where they require trading like for like / clout / reputation and providing new verified material. Monetary incentives are not what make those places tick.
If OAI's model can improve investigators' ability to wade through the ocean of depravity fast enough to rescue more kids, then it's a great thing.
Not everyone has the same ability to create content either, so requests can be made to other members in the community to nudity or create content according to some rubric.
If only.
The same "hard on crime" ratchet that happens to most every thing that's popular to punish and extremely unpopular to speak out against the tightening of said ratchet has happened here.
And yet, we have no means of disputing the accuracy of these inscrutable classifiers, nor any reason to think that similar approaches won't be applied to other "objectionable" content in other contexts, as is currently the fashion in Europe in the UK with regard to anything that could be conceivably deemed to be inciting hate, animus or disorder.
Automated technological surveillance and censorship is really, really having a moment right now. I'm not holding my breath that the pendulum swings the other way, but I sure hope it does.
Just don't put it in MY face.
With your argument, why would it any child pornography be immoral? It’s just 0s and 1s in the computer, innit? They never took the photo. They just downloaded it from the dark web.
Creating a real video of a real person getting their head chopped off: someone got killed, so that's clearly a problem. Creating a fake video depicting the same decapitation in graphic realism: no one bats an eye - go eat popcorn while it happens on a big screen and laugh about it.
So, why is one violent, illegal depiction given a green light, but another is reasoned about completely differently? I think there may be an argument, there, which is why I'm asking, but it doesn't spring to mind. How would you explain the contradiction to a peaceful alien species who uses asexual reproduction (knows nothing about sex) and is shocked and horrified by the violence that passes as entertainment to us?
I don't think the fake is legal if depicting the same act. One could be at least liable in a civil context.
> It is morally wrong because they depict children in sexual context. I mean these sickos come up with all kinds of things.
Can you tell me which ethical framework you are using to make your assertion? Right now it seems like an appeal to emotion, which is how we get things like the Patriot Act, KOSA, and all these id verification laws.
What ethical frameworks(s) is ”you shouldn’t do that, that’s wrong”?
You want me to think an act should be illegal as opposed to simply repugnant, show me harm to someone who was not a consenting adult. Addictive things should have safeguards around them to keep people from being addicted through not realizing the danger, but they shouldn't be prohibited.
Given the choice between someone doing actual child abuse and generating fake pictures on a computer, every time someone chooses the latter over the former, the world becomes a slightly better place.
You may argue that it would be even better to do neither, that's nice, it's called the Nirvana fallacy. That's already the law today, and people ignore it anyway.
Those who choices is a false dichotomy. It is so much more nuanced than that, and your framing is wrong. I take it you don’t work with sexual abuse and sexual offenders?
"So you don't think it's morally wrong to generate _______ using AI?"
"So you don't think it's morally wrong to _______ using AI?"
The matter can't be considered in isolation, and although anyone voicing a healthy fear of where this is going can be portrayed, through demagoguery, as "pro-CSAM" or "anti-children" at this juncture, the logical endpoint of this kind of surveillance is in far more unpalatable places.
As the good folks fighting--a losing battle, at present--for digital freedom and privacy know, guarding dearly held liberties means securing those liberties for criminals and antisocial pursuits, too. That's just the cost of freedom.
To elaborate, I was reading it to mean that they hoped the generation would be allowed because it doesn't actually involve a human.
What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?
I guess calling it a cost recognizes there is something wrong with it at least.
I don't have a great answer on whether it should be "allowed", but I see much wrong--as a matter of grave implications for freedom--with all the presently available and deployed techniques for "disallowing" it.
Im not asking about if it should be allowed, I'm asking if you find anything morally wrong about making child porn via AI, or consuming it.
Why can't you just say "yeah it's morally wrong?" Why is that not obvious?
Im acknowledging that hes concerned about unintended consequences. Now my question is if he finds anything morally wrong with ai generated child. A lack of an answer seems to indicate no but I’ll still entertain a more direct answer.
You could have just addressed his argument about side effects, but instead went on to the most nuclear toxic part of the discussion in order to get an own on him.
Do you think it is morally wrong to generate a violent image?
You misunderstand me. It's perfectly clear to me that AI generated child porn is immoral and should be illegal. So there is no "either."
That it's not clear -- whether AI generated child porn is immoral or not -- answers the question I posed to you by itself.
I personally am quite repulsed by child pornography, but am I prepared to universalise that for everyone when no child is being abused? I'm not sure. As I said in my long comment, I don't think there's a lot of precedent for that, outside of obscenity laws that apply to traditional visual art. We generally think of such laws to be on the wrong side of free speech today, though there are cross-cutting and contradictory trends in this area.
Beyond that, I am broadly supportive of electronic and digital freedom. There are very different kinds of things one can do with AI-generated erotic content, and there's a reason most prosecutions focus on distribution, revenge porn, blackmail, etc., rather than the mere fact of the generation _per se_. That aligns with my values. Maybe AI child porn is immoral, but telling people what visual concepts they can and cannot generate in the privacy of their own home for own gratification isn't moral either, to my way of thinking.
A number of European jurisdictions--and famously, the UK--do just that, and they aren't necessarily wrong. It is a tangible social good to keep the unthinkable cruelty of Nazi atrocities and the Holocaust alive in our collective and institutional memory. God forbid we ever forget these... actually, we seem to be well on our way, in that doom loop of history.
We haven't really had to confront, as a society, the ethics of photorealistic but utterly synthetic content, which can be generated quickly and at a very large scale, that contravenes our legal and social norms. There are clearly important nuances: purely synthetic content is not the same as deepfakes, which the courts have called "morphed" images.
I hear people who say it normalises sexual exploitation of children and makes us callous and indifferent to it. I hear people saying it's better than actual exploitation of real children, and that--reasoning from long-established free speech and privacy principles present elsewhere in our jurisprudence--the government shouldn't be telling people what they can enjoy in the privacy of their own home, provided that thing doesn't have a human victim, nor appears in any legally cognisable way to harm a class of victims (this is where much "hate speech" and "antisocial ideation" type laws dwell in the UK and EU).
I'm not entirely sure what to think. I don't have a great theory of mind for the kind of person who generates AI child porn. I'm sure many people think they do, but I can confidently say that I don't. Are they prone to real-life sex crimes against children? Is there a high degree of overlap with sundry groomers and predators? Is there a correlation between their interest in this and sexual abuse of their own family members? Their own children?
Or are they just curious techno-libertarians, maybe with a contrarian streak or a few unnerving kinks? What's a child in this context, anyway? Are we talking about toddlers or teenagers? Are they a child, or the kind of quasi-adult Judith Levine had in mind in Harmful to Minors (1994)? Is the perpetrator here more in the style of a 'predator', or just a 19 year-old horsing around? I don't really know what these folks are thinking or making, as I've never met anyone with such open proclivities. I'd guess, as with most things sexual, that many things are a lot more common than one might glean from the readily visible surface. I'd also guess that the truly depraved extreme stuff is at the margins. I think? I'm not a sex researcher.
Sex crime against children is a big dragnet, though: in my day, they put away Genarlow Wilson here in Georgia for 10 years:
https://en.wikipedia.org/wiki/Wilson_v._State
Most people seem to agree today that this was an instance of justice run amok and massively cruel and unusual punishment. The Georgia Supreme Court did, too, if only by 4-3. However, I was in high school and college in those days, and back in the day, advocating for this teen earned one few friends, and a "paedo" label, even though we all had girlfriends and boyfriends when we were in high school and ... I guess we were just supposed to pretend Genarlow Wilson was some apex sex predator because he was black, not sure what the message was. It's hard for me to imagine that a similar spectrum of possibilities, and shifting norms, doesn't exist in the world of child porn, even though it's an area that rarely sees the light of any kind of nuance. I refuse to believe it's just this opaque monolith, just as I refuse to believe the socialist bloc was one univocal monolith. There have got to be things that common sense can agree are somehow worse than other things, even if "better" is not a word one would reach for.
Anyway, my intent isn't to muddy the waters with "troubled" distinctions, but rather to say that without that kind of theory of who we're talking about, some details to enrich a mental picture and a moral penumbra, I can't really say if I think it's wrong. I don't know who's doing what in your example, nor the why. I'm reduced to mostly talking in the narrowly procedural mood of Anglo-American jurisprudence, which deals with rights and harms and boundaries and definitions, not, by and large, a central idea of the good. That works in court, which deals with whether one is allowed to do something, but doesn't settle the moral question much.
My first instincts are quite liberty-leaning on this matter, and in a world that is substantially electronic, digital freedom may matter more than any other kind, and we clearly took it for granted in the free and open Internet of decades past. I can appreciate the arguments from the other side, but they all strike me as special cases of much larger principles at stake--it's just that nobody wants to impale themselves on the child porn question due to its incendiary character.
Not true in terms of the law and philosophy. Congress passed a law already making child porn illegal for computer generated images of children.
In terms of social norms its also not novel. People have debated if pedophilia is morally wrong absent the harm to the child. Some people believe sex is right/wrong based on personal details like consent and harm and others factor in more natural order (such as roles - children, spouse etc). Maintaining integrity of roles makes it trivial to exclude incest and cases like this.
And the Supreme Court overturned it in Ashcroft vs. Free Speech Coalition
In a different time, place and era in technology, well before it was possible to create, at scale, lifelike and photorealistic images of almost anything, none of it incorporating real people. Back then, I think the idea was to criminalise well-done "Photoshop" jobs, or something like that.
The scale does matter. If computers, for example, become extensions of our imagination, so that we can wish a mental example into vivid, lifelike being with the touch of a button, and this becomes routine, then prohibiting a computer operation may tread, in many instances, close to prohibiting a category of expression, maybe even a category of thought (who knows how far the man-machine synthesis will go if Elon has his way...)
I don't have all the answers, I don't think Congress does, and I don't think you do.
Im asking you, do you find anything morally wrong with AI generated child porn?
The best I can do in trying to condense it is: "AI generated child porn" is hopelessly vague. What exactly do you mean?
If you're thinking, "what do you mean 'what do you mean?!'", please read my long comment.
Surely you can grant that child pornography does exist, right? That's what I'm asking about. I'll take you at your word that you do not know, but judging off what you said I can't see a single reason why you think maybe it's even a little bad.
I don't know if that's the turn law is ultimately going to take, but its hand may be forced by the impracticality of the alternative. That's the only reason any of us have any privacy at all: it's mostly a technological feasibility question. As someone said elsewhere in this discussion, the state would absolutely read our thoughts if it were easy for them to do so. As a corollary to that, I'd say that as technology has made electronic surveillance easier, the state and private companies have both helped themselves gluttonously, and we've barely scratched the surface as laypeople.
The morality of digital freedom and privacy is heavily entwined with my moral consideration of absolutely any digital transgression. When the underlying mechanism is purely digital, the question itself is digital. This is not a question about child porn to me. I take for granted people will have varying views on it. Instead, it's a question of, "Is there anything that is so wrong that one should not be allowed to privately do it with their computer in a way that doesn't straightforwardly affect anyone else?" At first instance, I'd say no.
As far as I understand the laws in e.g. Germany, this is not impossible.
The reason for that is you can prove the age of the real person to be 20, but you could not do that for the AI filter version because you have essentially erased the birth certificate.
This results in a problem: what is age in the context of fiction? The real world age of the character depends on the creation/publishing date, with all characters involved being the same real world age. If you go by the fictional age, then you run into the issue that the author can manipulate the fictional age.
The only way to enact the spirit of the law is to ban the functional consequence. What effect does the content have and where does it derive the effect from? The conclusion is that age must be reverse engineered based on the body shape, proportions, height, etc, but this means a child is no longer an age.
By treating fiction and reality the same, we have opened up pandora's box. Banning CSAM no longer means banning literal child porn, it means banning abstract concepts that may be present anywhere even if no children are involved.
For me the solution is obvious. Everything is its own crime. Purely fictional CSAM (normalization) is a separate crime from how non-fictional CSAM (harm to child) is a crime. Deepfake CSAM (harassment) is a crime but it is neither purely fictional nor non-fictional, so it is another crime again. You see, everything can be a crime and not be the same crime.
The biggest problems come from treating things that are not the same as if they are the same.
Why treat someone running a drug lab, someone selling a drug [0] and someone consuming the drug as if they are all the same? Because it's about total surveillance and not really about CSAM. Any demarcation line that sets a boundary goes against the total surveillance goal.
[0] which can be synthesized via AI without your own lab
In the past, the answer was more straightforward: child porn is visual depiction of children being used for sexual gratification. Notwithstanding rare, labour-intensive and not especially photo-realistic artistic depictions, as well as some ambiguous cases about artistic license, by and large, child porn depicted things that actually happened.
In the generative AI world, that's not true. At the very least, it forces a reckoning with the question: what is the prohibition of child porn really about? Is it about preventing real-world harm to children, or is it about preventing manifestations of the concept of sexualisation of children, on the theory that its mere existence harms children in principle?
This is a tendentious framing, but legally detailed:
https://enoughabuse.org/ai-csam-ruling-what-the-seventh-circ...
Specifically:
> The most important point is also the easiest to lose in the headlines: the AI-CSAM ruling in the Anderegg case did not hold that AI-generated CSAM is generally protected by the First Amendment.
If you think that's morally wrong, what if we change the subjects to an enthusiastically-consenting fifteen year old and sixteen year old? What if the author of the story is themselves fifteen? What if they're writing fiction about them and their girlfriend based on their intense and very enthusiastic mutual attraction, with the full knowledge and consent of their girlfriend?
If you think everything in the second paragraph is morally wrong, do you think all written smut, erotica, and pornography is morally wrong, or just those that depict humans under the age of eighteen; regardless of whether or not the authors are themselves under the age of eighteen? If the latter, why do you believe that it's morally wrong for teenage lovers to write sexy stories about each other, and do you believe that it's morally wrong for those lovers to verbally tell stories about their sexual escapades to their teenage friends?
[0] ...given the context, this would almost certainly be considered dubcon...
[1] For clarity, I very very strongly believe that the acts depicted in the hypothetical work of fiction in this paragraph are not things that should be done by any humans. I'm asking very specifically about the creation of the work of fiction.
I’d guess this is The difference between a substitute and complimentary good.
More cigarette ads didn’t just appeal to people who like cigarette ads, but smokers as well.
If the before is “only real” and the after is “real and generated”, why would real become more valuable? Supply is constant or higher, so the price should be constant or decrease.
It’s all types of porn, including deep fakes.
The effort to create content is negligible so everything has gone up.
Pornography is illegal to download or save in Singapore… I wonder how they were able to train a model without the data.
Datasets then could have been acquired by torrenting or scraping a porn streaming site, while never actually viewing the training set to verify (nor the test data for the model).
Or they just did something illegal, or have gotten some exemption. :)
/s
>A Bloom filter is a space-efficient, probabilistic data structure used to test whether an element is in a set. It can be “definitely not in the set” or “possibly in the set,” meaning false positives are possible but false negatives are not.
I deal with a lot of user-generated content, and I have been using OpenAI as a tool to help find offending content. Now I see that if CSAM ever makes it into one of my systems, OpenAI is going to send the authorities after me if their model sees it too, so that's a dealbreaker for that use case, and it was a big one!
It's also wild that they are trying to detect and report AI-generated CSAM. This does absolutely nothing to help protect actual children.
Say more. You have no problem with AI generated child porn?
But detecting purely AI-generated child porn requires something that actually scans your images and tries to determine what is in them. That is WAY more invasive than a fingerprint scan.
And, on top of that, targeting people for purely AI-generated child porn doesn't accomplish anything to protect children. It mostly just targets mentally ill people for incarceration. I don't think we handle this problem correctly as-is, but if we are going to be out there busting people for CP, I would prefer that we be focused on finding child abusers and bringing them to justice, not just rounding up people with problematic attractions. The only real exception is AI being used to "nudify" children. That's fucked up and targets real kids, but I don't know how you go about detecting this any less problematically than anything else.
At no point during this pursuit do I think it is appropriate for my AI agent to be investigating anything on my computer to report to the authorities, because there are plenty of use cases that involve incidental exposure to CSAM, and I don't need the cops coming to arrest me because I used Codex to clean up my social website's report queue, and it came across offending content.
And for the record, as someone who has had to clean social websites up, there is a huge difference between AI-generated CSAM, and real CSAM. The AI stuff is just gross, but the real stuff is traumatizing. It's a shame that AI isn't going to make this easier for me filter out, rather it just makes it riskier for me to use my computer at all.
If that is a worry you carry around, I really hope you work harder to make it never be the case lol. FYI it’s not a normal thing to worry about.
No, anyone that lets the public upload content will eventually have to deal with figuring out how to filter and remove objectionable content, ranging from copyrighted porn, copyrighted software, to CSAM.
AI companies should be providing workflows specifically for this, not shutting down accounts.
Being able to let an cold emotionless AI review all my users images, and just destroy the CSAM and forward it to NCMEC or whoever my AI provider partners with to make sure the data is captured and reported would be great. It could keep my website clear of illegal user content, keep child protection organizations and the authorities in-the-know, and prevent me from ever having to lay eyes on something that is going to give me nightmares.
It's been like that for the entire 20 years I have run these kinds of websites. It sucks, but it's just a sad reality of the world we live in.
I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.
How should society/law change in response? Is it purely a give and take between surveillance/right-to-not-be-watch and trying to prevent all bad things from happened before they happen?
Privacy, unfortunately, is largely the result of it being impractical to violate it.
I see zero reason to believe that states wont read your thoughts if they can easily manage to.
Dick's original story is actually much darker and more realistic in that the main character's motivation is as much about preserving the appearance of infallibility in the precrime system in the face of it's flaws as it is about proving his innocence.
Tech Company: At long last, we have created the Precrime Division from classic sci-fi novelette The Minority Report.
Most people think AI-generated child porn is wrong. But the consent/harm framework can't explain why AI generated child porn is wrong. Either you reach for indirect harms which can just be stipulated away (its not shared so police aren't confused, local model, etc.), or you have to argue in favor of AI generated child porn. In fact we have seen plenty of that here. And this framework has similar problems with necrophilia, bestiality, and adult incest.
A framework that allows for some natural order does not have this issue. If there is some natural order, and right/wrong are not just a function of preferences, you dont have to judge actions solely by effects. From that view, you can say sex has a proper place and purpose, and children have a proper role -- to be protected, raised, and not sexualized. Sexualizing a child violates that order whether or not a real child is ever involved. So the wrongness doesn't require any specific child to be harmed.
I think thats what most people already sense - that the wrongness was never only about harms to children.
I would agree that not every moral/immoral thing needs to be enforced by law. I do think AI generated child porn should be illegal but I take the broader point that not every immoral thing should be illegal.
So to that extent I already agree with you that people should be able to gravitate towards their own communities. I will gravitate towards the community that doesn't tolerate ai generated child porn. Will you?
https://developers.openai.com/api/docs/models/omni-moderatio...
Don't get me wrong: these are very necessary tools for the modern internet to function, and I'm sure _drastically_ reduce the review burden on humans, but the skeptic in me does not believe that OpenAI is being entirely altruistic here.
It requires all devices integrated into one single consciousness.
Could be a great start for "rogue AI"
Jokes aside, I'm glad they are giving tools to help fight this as currently your options are limited because every provider will simply return a content warning if you try to use an LLM as a moderator.
Remember when they wanted Apple to scan everybody’s devices locally?
I’m not sure what the solution is to deal with pedophiles but I’m sure it starts with parents monitoring their kids better and keeping young children from being isolated from real world social groups. I’m also sure it doesn’t involve using classifiers on a person’s own device to detect ‘possible’ child porn, which is where this is headed. Who can forget the father who had photos of his sick baby on Google Drive for his paediatrician to see and got reported to the police.
If it really is all about protecting kids then maybe the NCMEC should be lobbying to pass laws banning kids from the internet. Think about how many harms you could reduce instantly (bye bye Instagram, Snapchat, etc).
Similar to Roblox ignoring child grooming in their platform because they make money off of it. https://www.bloomberg.com/features/2024-roblox-pedophile-pro...
Similar to Facebook and other platforms feeding self-harm content to vulnerable children and teenagers, and even connecting them to dealers. These companies know their platforms are not safe for kids. https://www.bloomberg.com/features/cant-look-away/
Whatever you say about "non-offending" (and possessing child porn IS an offence in most places). I would definitely not have wanted you anywhere near my kids.
I believe AI is one of the only ways to engage with these desires in an ethical way.
Of course, you can always recommend abstinence, but unless you're rather conservative in your views, you probably don't think that's a good way to deal with teenage pregnancy, homosexuality, etc. So why would this be the exception?
Worth remembering, though, that the most likely offenders are family & friends. We spend a lot of time worrying about stranger danger because we cannot come to terms with just how common this predilection really is.
>In SafetyKit’s evaluations involving drawings or animations that sexualize children, the classifier detected 90% of cases
>This is the first CSAM classifier with broad availability that has a specific signal on indicating if CSAM may be AI generated.
What children are being abused by drawings or AI generated images, stories, etc?
I understand the cases where it's problematic to make such content based on real children and share it, and I think it's probably a bad idea to allow photorealistic child porn to be publicly posted on the internet for the sake of not confusing law enforcement (in the same way posting misinformation is harmful), but if these images are not realistic, or if it's just text, or if they are realistic but they aren't being shared on the internet, who is being harmed?
Do you also find these permissible, since there is no consent violation nor harm?
- Necrophilia
- Bestiality
- Adult incest
The problem with consent/harm framework is that you either have to start arguing some very indirect and abstract harm or you have to accept conclusions that everyone else thinks are just kind of fucked up to be consistent.
Once you introduce some form of natural law into your ethical framework it's trivial to say all these things are not permissible but I guess AI generated child porn and necrophilia is preferable to natural law to many.
Bestiality is obviously harmful to animals. Best case dubious if you assume animals could consent and would be willing, but seems like there's little evidence for that.
Adult incest is only harmful if it leads to pregnancy and birth defects.
Finding something gross doesn't mean it's immoral.
I can understand the arguments others have about AI generated child porn creating more demand, but I think we're rapidly approaching a world where human beings can engage in any kind of (what would otherwise be harmful) behavior in VR, assuming we don't all die from misaligned AI. I would prefer to live in a world where I can fulfill my desires in a harmless way, even if they're abhorrent to others, rather than a world of value lock-in based on antiquated views of harm.
In the scenario where the dead body doesn't have any loved ones whats wrong with necrophilia?
>Adult incest is only harmful if it leads to pregnancy and birth defects
And when it doesn't lead to pregnancy and birth defects? Are you OK with incest then?
Regarding incest, yes. I don't see the harm.
I understand the distinction between offending and non offending.
But pedophile is inclusive of both.